Re: [ANNOUNCE] Apache Ignite 2.7.6 Released
Alex, thanks for driving the release to the final milestone. This release is small but with crucial fixes. If anybody is on native persistence then schedule an upgrade. - Denis On Fri, Sep 20, 2019 at 3:25 AM Alexey Goncharuk wrote: > The Apache Ignite Community is pleased to announce the release of Apache > Ignite 2.7.6. > > Apache Ignite [1] is a memory-centric distributed database, caching, and > processing platform for transactional, analytical, and streaming workloads > delivering in-memory speeds at petabyte scale. > > This release introduces addresses frequent usability and critical > stability issues > https://ignite.apache.org/releases/2.7.6/release_notes.html > > Download the latest Ignite version from here: > https://ignite.apache.org/download.cgi > > Please let us know [2] if you encounter any problems. > > Regards, > Alexey Goncharuk on behalf of Apache Ignite Community > > [1] https://ignite.apache.org > [2] https://ignite.apache.org/community/resources.html >
Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]
I wrote about code freeze at December 18, 2019, ok, we can move one week earlier to 11 December Voting + Release could be after 10th January. пт, 20 сент. 2019 г. в 15:43, Maxim Muzafarov : > Alexey, > > It is not a problem to shift release a bit later or earlier, but I'm > strictly against having `code freeze` stage on holidays (the Christmas > holidays at the end of December and New Year holidays at the beginning > of January). From my point, it's better to have it completed `code > freeze` stage before December 23th or started after 10th January. > > Thoughts? > > > > On Fri, 20 Sep 2019 at 15:09, Dmitriy Pavlov wrote: > > > > +1 For Maxim as release manager. > > > > Maxim, > > > > It is a good thing that you have committer rights, and most of the steps > > you will be able to complete yourself. > > > > But please engage one from PMC member to complete steps from the release > > process where PMC rights are required > > https://cwiki.apache.org/confluence/display/IGNITE/Release+Process At > > least, access to docker and nuget creds requires PMC membership. > > > > Feel free to ping me, I will assist, as well. > > > > Sincerely, > > Dmitriy Pavlov > > > > > > пт, 20 сент. 2019 г. в 14:59, Alexey Zinoviev : > > > > > For Spark and ML components the best dates should be moved to one month > > > later, what's about? > > > There are a lot of features there, but a lot of bugs and minor > > > improvements in JIRA too > > > > > > Also I support you as a release manager > > > > > > Scope Freeze: December 4, 2019 > > > Code Freeze: December 18, 2019 > > > Voting Date: January 10, 2019 > > > Release Date: January 17, 2019 > > > > > > пт, 20 сент. 2019 г. в 14:44, Maxim Muzafarov : > > > > > > > Igniters, > > > > > > > > > > > > It's almost a year has passed since the last major Apache Ignite 2.7 > > > > has been released. We've accumulated a lot of performance > improvements > > > > and a lot of new features which are waiting for their release date. > > > > Here is my list of the most interesting things from my point since > the > > > > last major release: > > > > > > > > Service Grid, > > > > Monitoring, > > > > Recovery Read > > > > BLT auto-adjust, > > > > PDS compression, > > > > WAL page compression, > > > > Thin client: best effort affinity, > > > > Thin client: transactions support (not yet) > > > > SQL query history > > > > SQL statistics > > > > > > > > I think we should no longer wait and freeze the master branch anymore > > > > and prepare the next major release by the end of the year. > > > > > > > > > > > > I propose to discuss Time, Scope of Apache Ignite 2.8 release and > also > > > > I want to propose myself to be the release manager of the planning > > > > release. > > > > > > > > Scope Freeze: November 4, 2019 > > > > Code Freeze: November 18, 2019 > > > > Voting Date: December 10, 2019 > > > > Release Date: December 17, 2019 > > > > > > > > > > > > WDYT? > > > > > > > >
Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]
Alexey, It is not a problem to shift release a bit later or earlier, but I'm strictly against having `code freeze` stage on holidays (the Christmas holidays at the end of December and New Year holidays at the beginning of January). From my point, it's better to have it completed `code freeze` stage before December 23th or started after 10th January. Thoughts? On Fri, 20 Sep 2019 at 15:09, Dmitriy Pavlov wrote: > > +1 For Maxim as release manager. > > Maxim, > > It is a good thing that you have committer rights, and most of the steps > you will be able to complete yourself. > > But please engage one from PMC member to complete steps from the release > process where PMC rights are required > https://cwiki.apache.org/confluence/display/IGNITE/Release+Process At > least, access to docker and nuget creds requires PMC membership. > > Feel free to ping me, I will assist, as well. > > Sincerely, > Dmitriy Pavlov > > > пт, 20 сент. 2019 г. в 14:59, Alexey Zinoviev : > > > For Spark and ML components the best dates should be moved to one month > > later, what's about? > > There are a lot of features there, but a lot of bugs and minor > > improvements in JIRA too > > > > Also I support you as a release manager > > > > Scope Freeze: December 4, 2019 > > Code Freeze: December 18, 2019 > > Voting Date: January 10, 2019 > > Release Date: January 17, 2019 > > > > пт, 20 сент. 2019 г. в 14:44, Maxim Muzafarov : > > > > > Igniters, > > > > > > > > > It's almost a year has passed since the last major Apache Ignite 2.7 > > > has been released. We've accumulated a lot of performance improvements > > > and a lot of new features which are waiting for their release date. > > > Here is my list of the most interesting things from my point since the > > > last major release: > > > > > > Service Grid, > > > Monitoring, > > > Recovery Read > > > BLT auto-adjust, > > > PDS compression, > > > WAL page compression, > > > Thin client: best effort affinity, > > > Thin client: transactions support (not yet) > > > SQL query history > > > SQL statistics > > > > > > I think we should no longer wait and freeze the master branch anymore > > > and prepare the next major release by the end of the year. > > > > > > > > > I propose to discuss Time, Scope of Apache Ignite 2.8 release and also > > > I want to propose myself to be the release manager of the planning > > > release. > > > > > > Scope Freeze: November 4, 2019 > > > Code Freeze: November 18, 2019 > > > Voting Date: December 10, 2019 > > > Release Date: December 17, 2019 > > > > > > > > > WDYT? > > > > >
[jira] [Created] (IGNITE-12207) Inclusion of super.toString() info into some descenders of GridCacheMessage
Dmitriy Sorokin created IGNITE-12207: Summary: Inclusion of super.toString() info into some descenders of GridCacheMessage Key: IGNITE-12207 URL: https://issues.apache.org/jira/browse/IGNITE-12207 Project: Ignite Issue Type: Improvement Affects Versions: 2.7.6, 2.7 Reporter: Dmitriy Sorokin Assignee: Dmitriy Sorokin -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]
+1 For Maxim as release manager. Maxim, It is a good thing that you have committer rights, and most of the steps you will be able to complete yourself. But please engage one from PMC member to complete steps from the release process where PMC rights are required https://cwiki.apache.org/confluence/display/IGNITE/Release+Process At least, access to docker and nuget creds requires PMC membership. Feel free to ping me, I will assist, as well. Sincerely, Dmitriy Pavlov пт, 20 сент. 2019 г. в 14:59, Alexey Zinoviev : > For Spark and ML components the best dates should be moved to one month > later, what's about? > There are a lot of features there, but a lot of bugs and minor > improvements in JIRA too > > Also I support you as a release manager > > Scope Freeze: December 4, 2019 > Code Freeze: December 18, 2019 > Voting Date: January 10, 2019 > Release Date: January 17, 2019 > > пт, 20 сент. 2019 г. в 14:44, Maxim Muzafarov : > > > Igniters, > > > > > > It's almost a year has passed since the last major Apache Ignite 2.7 > > has been released. We've accumulated a lot of performance improvements > > and a lot of new features which are waiting for their release date. > > Here is my list of the most interesting things from my point since the > > last major release: > > > > Service Grid, > > Monitoring, > > Recovery Read > > BLT auto-adjust, > > PDS compression, > > WAL page compression, > > Thin client: best effort affinity, > > Thin client: transactions support (not yet) > > SQL query history > > SQL statistics > > > > I think we should no longer wait and freeze the master branch anymore > > and prepare the next major release by the end of the year. > > > > > > I propose to discuss Time, Scope of Apache Ignite 2.8 release and also > > I want to propose myself to be the release manager of the planning > > release. > > > > Scope Freeze: November 4, 2019 > > Code Freeze: November 18, 2019 > > Voting Date: December 10, 2019 > > Release Date: December 17, 2019 > > > > > > WDYT? > > >
Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]
For Spark and ML components the best dates should be moved to one month later, what's about? There are a lot of features there, but a lot of bugs and minor improvements in JIRA too Also I support you as a release manager Scope Freeze: December 4, 2019 Code Freeze: December 18, 2019 Voting Date: January 10, 2019 Release Date: January 17, 2019 пт, 20 сент. 2019 г. в 14:44, Maxim Muzafarov : > Igniters, > > > It's almost a year has passed since the last major Apache Ignite 2.7 > has been released. We've accumulated a lot of performance improvements > and a lot of new features which are waiting for their release date. > Here is my list of the most interesting things from my point since the > last major release: > > Service Grid, > Monitoring, > Recovery Read > BLT auto-adjust, > PDS compression, > WAL page compression, > Thin client: best effort affinity, > Thin client: transactions support (not yet) > SQL query history > SQL statistics > > I think we should no longer wait and freeze the master branch anymore > and prepare the next major release by the end of the year. > > > I propose to discuss Time, Scope of Apache Ignite 2.8 release and also > I want to propose myself to be the release manager of the planning > release. > > Scope Freeze: November 4, 2019 > Code Freeze: November 18, 2019 > Voting Date: December 10, 2019 > Release Date: December 17, 2019 > > > WDYT? >
Apache Ignite 2.8 RELEASE [Time, Scope, Manager]
Igniters, It's almost a year has passed since the last major Apache Ignite 2.7 has been released. We've accumulated a lot of performance improvements and a lot of new features which are waiting for their release date. Here is my list of the most interesting things from my point since the last major release: Service Grid, Monitoring, Recovery Read BLT auto-adjust, PDS compression, WAL page compression, Thin client: best effort affinity, Thin client: transactions support (not yet) SQL query history SQL statistics I think we should no longer wait and freeze the master branch anymore and prepare the next major release by the end of the year. I propose to discuss Time, Scope of Apache Ignite 2.8 release and also I want to propose myself to be the release manager of the planning release. Scope Freeze: November 4, 2019 Code Freeze: November 18, 2019 Voting Date: December 10, 2019 Release Date: December 17, 2019 WDYT?
Re: [IEP-35] Monitoring & Profiling. Phase 2
Hello, Alex. Good catch, thank you. I will add enabling of JMX and SQL exporters for system views, by default. В Ср, 18/09/2019 в 16:09 +0300, Alex Plehanov пишет: > One more point to discuss: Wouldn't it be better to have enabled system > views by default? > To enable views admin must restart the node, sometimes it's an issue. > Views cost almost nothing in terms of performance until they are explicitly > requested, so is their a reason to disable views by default? > > вт, 17 сент. 2019 г. в 12:48, Alexey Goncharuk : > > > Folks, > > > > I honestly tried to follow the discussion, but I think that I lost the > > point of the debate. Should we try to exploit the newly introduced slack to > > discuss the change and then send a follow-up here? > > > > --AG > > signature.asc Description: This is a digitally signed message part
Re: [ANNOUNCE] Apache Ignite 2.7.6 Released
Great, go 2.8! пт, 20 сент. 2019 г. в 13:25, Alexey Goncharuk : > The Apache Ignite Community is pleased to announce the release of Apache > Ignite 2.7.6. > > Apache Ignite [1] is a memory-centric distributed database, caching, and > processing platform for transactional, analytical, and streaming workloads > delivering in-memory speeds at petabyte scale. > > This release introduces addresses frequent usability and critical stability > issues > https://ignite.apache.org/releases/2.7.6/release_notes.html > > Download the latest Ignite version from here: > https://ignite.apache.org/download.cgi > > Please let us know [2] if you encounter any problems. > > Regards, > Alexey Goncharuk on behalf of Apache Ignite Community > > [1] https://ignite.apache.org > [2] https://ignite.apache.org/community/resources.html >
Re: TDE Master key rotation (Phase-2)
Nikolay, you are right in many ways. I updated the design on the wiki. [1] [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652381 пт, 20 сент. 2019 г. в 13:49, Nikolay Izhikov : > > Nikita > > > I suggested the implementation where the encryption manager is > > responsible for storing the master key id. > > I don't think it's a right proposal. > > 1. EncryptionSpi implementation becomes more complicated. Developer of it > should be aware of Ignite deployment scenarious, etc. > Imagine implementation when EncryptionSpi send master key id to some external > storage over network(it's happen in Discovery thread) > > 2. Implementation would be duplicate features(saving master key id) > > 3. We already store cache keys in metastore. For me it's a native approach to > store master key id in the same place. > > What do you think? > > > В Пт, 20/09/2019 в 13:39 +0300, Nikita Amelchev пишет: > > Nikolay, > > > > because I suggested the implementation where the encryption manager is > > responsible for storing the master key id. > > To implement this logic in the EncryptionSpi, we will need to > > introduce the methods look like this: > > > > setMasterKeyId(String masterKeyId) // Sets "current" master key id > > String getMasterKeyId() // Gets "current" master key id > > > > Follow methods will work with master key that setted by previous method: > > > > byte[] masterKeyDigest() > > byte[] encryptKey(Serializable key) > > Serializable decryptKey(byte[] key) > > > > If such implementation is more reasonable, I will do so. > > > > пт, 20 сент. 2019 г. в 13:04, Nikolay Izhikov : > > > > > > Why do we need "defaultMasterKeyId" instead of *current* master key id > > > that can be obtained with `KeystoreEncryptionSpi#getMasterKeyName()`? > > > > > > В Пт, 20/09/2019 в 12:56 +0300, Nikita Amelchev пишет: > > > > Nikolay, > > > > > > > > Thanks for the proposal, I like it. > > > > > > > > The GridEncryptionManager will control the process of master key > > > > rotation, so we should provide him master key id at startup. Seems we > > > > should get it from some configuration for encryption. > > > > > > > > I suggest just adding the String defaultMasterKeyId() method into the > > > > EncryptionSpi interface. This method gets default master key id used > > > > on first cluster start. > > > > > > > > The specific implementation will be responsible for setting this value. > > > > > > > > What do you think? > > > > > > > > пт, 20 сент. 2019 г. в 10:44, Nikolay Izhikov : > > > > > > > > > > Hello, Nikita > > > > > > > > > > > IgniteConfiguration: New methods will be added to the > > > > > > IgniteConfiguration: > > > > > > public IgniteConfiguration setEncryptionMasterKeyId(String > > > > > > masterKeyId) - sets master key id. > > > > > > public String getEncryptionMasterKeyId() > > > > > > > > > > We don't need it in the IgniteConfiguration. > > > > > > > > > > As you may know, we already have > > > > > KeystoreEncryptionSpi#setMasterKeyName. > > > > > Seems, we should add it to the EncryptionSpi itself. > > > > > > > > > > > > > > > В Ср, 18/09/2019 в 22:25 +0300, Nikita Amelchev пишет: > > > > > > Nikolay, thanks for participating. > > > > > > > > > > > > I have supplemented the design and clarify these moments. [1] > > > > > > > > > > > > [1] > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652381 > > > > > > > > > > > > ср, 18 сент. 2019 г. в 16:48, Nikolay Izhikov : > > > > > > > > > > > > > > Hello, Nikita. > > > > > > > > > > > > > > Thanks for starting this discussion. > > > > > > > > > > > > > > 1. We should add prerequisites for "master key rotation process" > > > > > > > in design. > > > > > > > Seems, it should be, "New master key available to EncryptionSPI > > > > > > > for each server node". > > > > > > > > > > > > > > 2. Please, use code formatting in wiki. It's make reading easier. > > > > > > > > > > > > > > 3. Please, clarify java API proposal. What will be changed and > > > > > > > how. > > > > > > > AFAIK we need to change EncryptionSPI, this should be covered in > > > > > > > design. > > > > > > > > > > > > > > 4. Please, clarify new CLI commands. > > > > > > > AFAIK we should have 2 command: > > > > > > > > > > > > > > 1. Start regular master key rotation process. > > > > > > > 2. Start local master key rotation process during node > > > > > > > recovery(for the case when key changed while node was down). > > > > > > > > > > > > > > В Ср, 18/09/2019 в 16:09 +0300, Nikita Amelchev пишет: > > > > > > > > Hi, Igniters. > > > > > > > > > > > > > > > > I'm going to implement the ability to rotate the master > > > > > > > > encryption key > > > > > > > > (TDE Phase 2). [1] > > > > > > > > Master key rotation required in case of it compromising or at > > > > > > > > the end > > > > > > > > of crypto period(key validity period). I prepared the design. > > > > > > > > [2] > > > > > > > > > > > > > > > > In briefly, master keys
Re: TDE Master key rotation (Phase-2)
Nikolay, because I suggested the implementation where the encryption manager is responsible for storing the master key id. To implement this logic in the EncryptionSpi, we will need to introduce the methods look like this: setMasterKeyId(String masterKeyId) // Sets "current" master key id String getMasterKeyId() // Gets "current" master key id Follow methods will work with master key that setted by previous method: byte[] masterKeyDigest() byte[] encryptKey(Serializable key) Serializable decryptKey(byte[] key) If such implementation is more reasonable, I will do so. пт, 20 сент. 2019 г. в 13:04, Nikolay Izhikov : > > Why do we need "defaultMasterKeyId" instead of *current* master key id that > can be obtained with `KeystoreEncryptionSpi#getMasterKeyName()`? > > В Пт, 20/09/2019 в 12:56 +0300, Nikita Amelchev пишет: > > Nikolay, > > > > Thanks for the proposal, I like it. > > > > The GridEncryptionManager will control the process of master key > > rotation, so we should provide him master key id at startup. Seems we > > should get it from some configuration for encryption. > > > > I suggest just adding the String defaultMasterKeyId() method into the > > EncryptionSpi interface. This method gets default master key id used > > on first cluster start. > > > > The specific implementation will be responsible for setting this value. > > > > What do you think? > > > > пт, 20 сент. 2019 г. в 10:44, Nikolay Izhikov : > > > > > > Hello, Nikita > > > > > > > IgniteConfiguration: New methods will be added to the > > > > IgniteConfiguration: > > > > public IgniteConfiguration setEncryptionMasterKeyId(String masterKeyId) > > > > - sets master key id. > > > > public String getEncryptionMasterKeyId() > > > > > > We don't need it in the IgniteConfiguration. > > > > > > As you may know, we already have KeystoreEncryptionSpi#setMasterKeyName. > > > Seems, we should add it to the EncryptionSpi itself. > > > > > > > > > В Ср, 18/09/2019 в 22:25 +0300, Nikita Amelchev пишет: > > > > Nikolay, thanks for participating. > > > > > > > > I have supplemented the design and clarify these moments. [1] > > > > > > > > [1] > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652381 > > > > > > > > ср, 18 сент. 2019 г. в 16:48, Nikolay Izhikov : > > > > > > > > > > Hello, Nikita. > > > > > > > > > > Thanks for starting this discussion. > > > > > > > > > > 1. We should add prerequisites for "master key rotation process" in > > > > > design. > > > > > Seems, it should be, "New master key available to EncryptionSPI for > > > > > each server node". > > > > > > > > > > 2. Please, use code formatting in wiki. It's make reading easier. > > > > > > > > > > 3. Please, clarify java API proposal. What will be changed and how. > > > > > AFAIK we need to change EncryptionSPI, this should be covered in > > > > > design. > > > > > > > > > > 4. Please, clarify new CLI commands. > > > > > AFAIK we should have 2 command: > > > > > > > > > > 1. Start regular master key rotation process. > > > > > 2. Start local master key rotation process during node > > > > > recovery(for the case when key changed while node was down). > > > > > > > > > > В Ср, 18/09/2019 в 16:09 +0300, Nikita Amelchev пишет: > > > > > > Hi, Igniters. > > > > > > > > > > > > I'm going to implement the ability to rotate the master encryption > > > > > > key > > > > > > (TDE Phase 2). [1] > > > > > > Master key rotation required in case of it compromising or at the > > > > > > end > > > > > > of crypto period(key validity period). I prepared the design. [2] > > > > > > > > > > > > In briefly, master keys will be identified by String masterKeyId. > > > > > > The > > > > > > concept of the masterKeyId will be added to the cache keys > > > > > > encryption > > > > > > process in EncryptionSpi. > > > > > > > > > > > > Users can configure master key id in IgniteConfiguration and will be > > > > > > able to manage the key rotation process from java API, JMX, CLI: > > > > > > - ignite.encryption().changeMasterKey(String masterKeyId) - starts > > > > > > master key rotation process. > > > > > > - String ignite.encryption().getMasterKeyId() - gets current > > > > > > master key id. > > > > > > > > > > > > Any thoughts? > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12186 > > > > > > [2] > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652381 > > > > > > > > > > > > > > > > > > > > > > > > -- Best wishes, Amelchev Nikita
[ANNOUNCE] Apache Ignite 2.7.6 Released
The Apache Ignite Community is pleased to announce the release of Apache Ignite 2.7.6. Apache Ignite [1] is a memory-centric distributed database, caching, and processing platform for transactional, analytical, and streaming workloads delivering in-memory speeds at petabyte scale. This release introduces addresses frequent usability and critical stability issues https://ignite.apache.org/releases/2.7.6/release_notes.html Download the latest Ignite version from here: https://ignite.apache.org/download.cgi Please let us know [2] if you encounter any problems. Regards, Alexey Goncharuk on behalf of Apache Ignite Community [1] https://ignite.apache.org [2] https://ignite.apache.org/community/resources.html
[jira] [Created] (IGNITE-12206) Partition state validation warns are not printed to log
Stepachev Maksim created IGNITE-12206:
-
Summary: Partition state validation warns are not printed to log
Key: IGNITE-12206
URL: https://issues.apache.org/jira/browse/IGNITE-12206
Project: Ignite
Issue Type: Bug
Reporter: Stepachev Maksim
Assignee: Stepachev Maksim
GridDhtPartitionsExchangeFuture.java
{{}}
{code:java}
if (grpCtx == null
|| grpCtx.config().isReadThrough()
|| grpCtx.config().isWriteThrough()
|| grpCtx.config().getCacheStoreFactory() != null
|| grpCtx.config().getRebalanceDelay() == -1
|| grpCtx.config().getRebalanceMode() ==
CacheRebalanceMode.NONE
|| grpCtx.config().getExpiryPolicyFactory() == null
|| SKIP_PARTITION_SIZE_VALIDATION)
return null;{code}
{{}}
Looks like a typo, probably it should be
grpCtx.config().getExpiryPolicyFactory() != null
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-12205) GridCachePartitionedSetWithClientSelfTest.testMultithreaded has 95,5% fail rate for long time
Stepachev Maksim created IGNITE-12205: - Summary: GridCachePartitionedSetWithClientSelfTest.testMultithreaded has 95,5% fail rate for long time Key: IGNITE-12205 URL: https://issues.apache.org/jira/browse/IGNITE-12205 Project: Ignite Issue Type: Bug Reporter: Stepachev Maksim Assignee: Stepachev Maksim This is well-know failure, need to investigate and fix. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: TDE Master key rotation (Phase-2)
Why do we need "defaultMasterKeyId" instead of *current* master key id that can be obtained with `KeystoreEncryptionSpi#getMasterKeyName()`? В Пт, 20/09/2019 в 12:56 +0300, Nikita Amelchev пишет: > Nikolay, > > Thanks for the proposal, I like it. > > The GridEncryptionManager will control the process of master key > rotation, so we should provide him master key id at startup. Seems we > should get it from some configuration for encryption. > > I suggest just adding the String defaultMasterKeyId() method into the > EncryptionSpi interface. This method gets default master key id used > on first cluster start. > > The specific implementation will be responsible for setting this value. > > What do you think? > > пт, 20 сент. 2019 г. в 10:44, Nikolay Izhikov : > > > > Hello, Nikita > > > > > IgniteConfiguration: New methods will be added to the IgniteConfiguration: > > > public IgniteConfiguration setEncryptionMasterKeyId(String masterKeyId) - > > > sets master key id. > > > public String getEncryptionMasterKeyId() > > > > We don't need it in the IgniteConfiguration. > > > > As you may know, we already have KeystoreEncryptionSpi#setMasterKeyName. > > Seems, we should add it to the EncryptionSpi itself. > > > > > > В Ср, 18/09/2019 в 22:25 +0300, Nikita Amelchev пишет: > > > Nikolay, thanks for participating. > > > > > > I have supplemented the design and clarify these moments. [1] > > > > > > [1] > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652381 > > > > > > ср, 18 сент. 2019 г. в 16:48, Nikolay Izhikov : > > > > > > > > Hello, Nikita. > > > > > > > > Thanks for starting this discussion. > > > > > > > > 1. We should add prerequisites for "master key rotation process" in > > > > design. > > > > Seems, it should be, "New master key available to EncryptionSPI for > > > > each server node". > > > > > > > > 2. Please, use code formatting in wiki. It's make reading easier. > > > > > > > > 3. Please, clarify java API proposal. What will be changed and how. > > > > AFAIK we need to change EncryptionSPI, this should be covered in design. > > > > > > > > 4. Please, clarify new CLI commands. > > > > AFAIK we should have 2 command: > > > > > > > > 1. Start regular master key rotation process. > > > > 2. Start local master key rotation process during node > > > > recovery(for the case when key changed while node was down). > > > > > > > > В Ср, 18/09/2019 в 16:09 +0300, Nikita Amelchev пишет: > > > > > Hi, Igniters. > > > > > > > > > > I'm going to implement the ability to rotate the master encryption key > > > > > (TDE Phase 2). [1] > > > > > Master key rotation required in case of it compromising or at the end > > > > > of crypto period(key validity period). I prepared the design. [2] > > > > > > > > > > In briefly, master keys will be identified by String masterKeyId. The > > > > > concept of the masterKeyId will be added to the cache keys encryption > > > > > process in EncryptionSpi. > > > > > > > > > > Users can configure master key id in IgniteConfiguration and will be > > > > > able to manage the key rotation process from java API, JMX, CLI: > > > > > - ignite.encryption().changeMasterKey(String masterKeyId) - starts > > > > > master key rotation process. > > > > > - String ignite.encryption().getMasterKeyId() - gets current master > > > > > key id. > > > > > > > > > > Any thoughts? > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12186 > > > > > [2] > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652381 > > > > > > > > > > > > > > > > > signature.asc Description: This is a digitally signed message part
Re: [IEP-35] Metrics management in Ignite
Hello, Andrey.
Thanks for starting this discussion.
> 1. There is no unified approach to adding metrics during development.
Yes, we have it.
Just call `MetricRegistry#longMetric` or other method and you got your counter.
> Now the logic is spread throughout the code base and there is now any
> way to find proper place where particular metric could be added to the
> registry
You should obtain your counter in the place where it be updated or exported.
If you update the same counter in several place, it seems to be as a bad code
design which should be investigated.
I treat this as a feature that adds flexibility to the Ignite code.
> Moreover, we can create registry in one place and add some
> set of metrics to the registry in another place and we have many
> examples of it.
What's wrong with this approach?
I treat this as a feature, also.
> Of course, metrics API allows it and it is meaningful drawback from my point
> view
The goal of Metric API was simplifying of metric creation.
Because, in the past years Ignite doesn't have a way to create meaningfull
metric.
I know many cases where specific metrics was not implemented because of
development complexity.
From my point of view, we shouldn't complicate process of creation of specific
metric.
It will slow down creation of new metric therefore slow down product evolving.
> 2. Introduce MetricRegistryBuilder that returns immutable instance of
> MetricRegistry
If we want immutable MetricRegistry we can create simple POJO objects with
metric getters as follows
public class CacheMetric {
public LongMetric putsCount() { ... }
public LongMetric getsCount() { ... }
}
and export it using `AttributeWalker` approach I implement in System View
contribution.
There is no need for a HashMap unified solution.
We can extend `AttibuteWalkerGenerator` to generate all required boilerplace
code (for enabling/disabling metrics, etc.)
I want to bold my statement: We should keep metric creation as simple as we can.
Developer of specific metric shouldn't change dozens of static interfaces,
implementations, enable/disable methods.
It should only update metric in the specific places in core code.
В Пн, 26/08/2019 в 20:36 +0300, Andrey Gura пишет:
> Hi, Igniters!
>
> I'm working on some metrics related aspects and it seems that we have
> missed some points regarding the metrics management. There are at
> least two major problems from my point of view.
>
> Problem definition.
> ---
>
> 1. There is no unified approach to adding metrics during development.
>
> It would be grate to have one common place where developer should add
> new metrics or can find answer for question "What kind of metrics do
> we have for smth?". I talk only about Ignite internal metrics, not
> user's metrics.
>
> Now the logic is spread throughout the code base and there is now any
> way to find proper place where particular metric could be added to the
> registry. Moreover, we can create registry in one place and add some
> set of metrics to the registry in another place and we have many
> examples of it.
>
> Of course, metrics API allows it and it is meaningful drawback from my
> point view. MetricRegistry is mutable and could be modified at any
> place of the code and any time of program execution. It allows
> developer follow the simplest way which is usually wrong. Also
> GridMetricManager.registry() method has two responsibilities: it
> creates and lookups registries. When I see registry() method call in
> the code I can't assume what exactly intention had developer (is it
> first creation of registry or modification of existing registry).
>
> 2. There is no unified approach to enabling/disabling metrics.
>
> Here I mean both problems: runtime enabling/disabling and code design
> that should allow do it in proper way. There is JIRA issue [1] about
> it.
>
> Now some metrics can be enabled/disabled in static configuration and
> at runtime (cache metrics for example) and other don't have such
> functionality. It should be unified. Any metrics set (eq.
> MetricRegistry in current implementation) should have one unified
> approach for enabling/disabling. Also we should provide infrastructure
> for it as part of metrics framework.
>
>
> Proposed solution.
> ---
>
> 1. Introduce new interface, e.g. MetriсSource that is responsible for
> metrics life cycle for one metrics source (e.g. cache metrics, system
> metrics, etc.) Each component with metrics should register own
> implementation of this interface in metrics manager
> (GridMetricManager).
>
> Interface declares the following methods:
>
> - String name() - returns sources name that will be available in
> special MBean responsible for access for metrics manager.
>
> - MetricRegistry enable() - returns MetricRegistry instance with
> allready registered metrics. Metric registry has the same name as name
> returned by MetricSource.name() method. This
[jira] [Created] (IGNITE-12204) In binary distribution, essential dependencies for ignite-spark missing
Ilya Kasnacheev created IGNITE-12204:
Summary: In binary distribution, essential dependencies for
ignite-spark missing
Key: IGNITE-12204
URL: https://issues.apache.org/jira/browse/IGNITE-12204
Project: Ignite
Issue Type: Improvement
Components: spark
Affects Versions: 2.7.6
Reporter: Ilya Kasnacheev
It seems that we only put direct dependencies of other JARs in our binary
distribution, and not transient ones.
For example, libs/optional/ignite-spark lacks the essential commons-lang3 jar,
which will lead to the following error immediately:
{code}
Exception in thread "main" java.lang.NoClassDefFoundError:
org.apache.commons.lang3.SystemUtils
at org.apache.spark.util.Utils$.(Utils.scala:1915)
at org.apache.spark.util.Utils$.(Utils.scala)
at
org.apache.spark.SparkConf.loadFromSystemProperties(SparkConf.scala:75)
{code}
It's almost impossible to fix without resorting to Maven source build.
I understand that adding Spark module to Ignite server is something not widely
used, but if we ship this module at all, we should make sure that it is usable
in some form.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: TDE Master key rotation (Phase-2)
Hello, Nikita > IgniteConfiguration: New methods will be added to the IgniteConfiguration: > public IgniteConfiguration setEncryptionMasterKeyId(String masterKeyId) - > sets master key id. > public String getEncryptionMasterKeyId() We don't need it in the IgniteConfiguration. As you may know, we already have KeystoreEncryptionSpi#setMasterKeyName. Seems, we should add it to the EncryptionSpi itself. В Ср, 18/09/2019 в 22:25 +0300, Nikita Amelchev пишет: > Nikolay, thanks for participating. > > I have supplemented the design and clarify these moments. [1] > > [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652381 > > ср, 18 сент. 2019 г. в 16:48, Nikolay Izhikov : > > > > Hello, Nikita. > > > > Thanks for starting this discussion. > > > > 1. We should add prerequisites for "master key rotation process" in design. > > Seems, it should be, "New master key available to EncryptionSPI for each > > server node". > > > > 2. Please, use code formatting in wiki. It's make reading easier. > > > > 3. Please, clarify java API proposal. What will be changed and how. > > AFAIK we need to change EncryptionSPI, this should be covered in design. > > > > 4. Please, clarify new CLI commands. > > AFAIK we should have 2 command: > > > > 1. Start regular master key rotation process. > > 2. Start local master key rotation process during node recovery(for > > the case when key changed while node was down). > > > > В Ср, 18/09/2019 в 16:09 +0300, Nikita Amelchev пишет: > > > Hi, Igniters. > > > > > > I'm going to implement the ability to rotate the master encryption key > > > (TDE Phase 2). [1] > > > Master key rotation required in case of it compromising or at the end > > > of crypto period(key validity period). I prepared the design. [2] > > > > > > In briefly, master keys will be identified by String masterKeyId. The > > > concept of the masterKeyId will be added to the cache keys encryption > > > process in EncryptionSpi. > > > > > > Users can configure master key id in IgniteConfiguration and will be > > > able to manage the key rotation process from java API, JMX, CLI: > > > - ignite.encryption().changeMasterKey(String masterKeyId) - starts > > > master key rotation process. > > > - String ignite.encryption().getMasterKeyId() - gets current master key > > > id. > > > > > > Any thoughts? > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12186 > > > [2] > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652381 > > > > > > signature.asc Description: This is a digitally signed message part
Re: Spark examples on TC
Hello, Denis. Spark uses Scala 2.11 [1] Scala 2.11 compatible with the jdk version described on [2]. Few notes from the link: > We recommend using Java 8 for compiling Scala code > JDK 11 compatibility notes > As of Scala 2.13.0, 2.12.8 and 2.11.12, JDK 11 support is incomplete I also got several issues with the Spark itself when trying to run it on the Java 11. Please, take a look at [3], [4] All in all, with the spark 2.3 we should stay on java8 to get things worked. [1] http://spark.apache.org/docs/2.4.0/ [2] https://docs.scala-lang.org/overviews/jdk-compatibility/overview.html [3] https://stackoverflow.com/questions/49961991/java-lang-illegalargumentexception-at-org-apache-xbean-asm5-classreader-initu?noredirect=1=1 [4] https://stackoverflow.com/questions/51352591/spark-java-illegalargumentexception-at-org-apache-xbean-asm5-classreader чт, 19 сент. 2019 г. в 21:39, Denis Magda : > Nikolay, > > What needs to be done to ensure Java 9++ works with Spark Examples? Do we > need to create another Spark module or drop some scala version? > > - > Denis > > > On Thu, Sep 19, 2019 at 9:23 AM Nikolay Izhikov > wrote: > > > Hello, Igniters. > > > > Examples suite on TC doesn't work correctly under java9+. [1], [2] > > > > The reason is the Spark examples. > > > > Spark 2.3(our current version) work with scala 2.11 and scala 2.11 > support > > of java9+ is "incomplete". > > > > Can we stick example suite to jdk8 only? > > > > [1] > > > https://ci.ignite.apache.org/viewLog.html?buildId=4610507=IgniteTests24Java8_Examples > > [2] > > > https://ci.ignite.apache.org/viewLog.html?buildId=4609013=IgniteTests24Java8_Examples > > >
