svn commit: r1812781 - in /ignite/site/trunk: events.html images/Ignite_tm_Logo_black_RGB.svg images/Ignite_tm_Logo_white_RGB.svg includes/footer.html includes/header.html index.html
Author: dmagda Date: Fri Oct 20 19:01:17 2017 New Revision: 1812781 URL: http://svn.apache.org/viewvc?rev=1812781&view=rev Log: updated ignite logos Added: ignite/site/trunk/images/Ignite_tm_Logo_black_RGB.svg ignite/site/trunk/images/Ignite_tm_Logo_white_RGB.svg Modified: ignite/site/trunk/events.html ignite/site/trunk/includes/footer.html ignite/site/trunk/includes/header.html ignite/site/trunk/index.html Modified: ignite/site/trunk/events.html URL: http://svn.apache.org/viewvc/ignite/site/trunk/events.html?rev=1812781&r1=1812780&r2=1812781&view=diff == --- ignite/site/trunk/events.html (original) +++ ignite/site/trunk/events.html Fri Oct 20 19:01:17 2017 @@ -117,6 +117,37 @@ under the License. + +Apache Spark and Apache Ignite: Where Fast Data Meets the IoT + + +Dublin Spark Meetup, Speaker - Akmal Chaudhri +October 26, 2017 + +In this session, Akmal will show attendees how to build a Fast Data solution that will receive endless streams from the +IoT side and will be capable of processing the streams in real-time using Apache Ignite's cluster resources. In particular, +attendees will learn about data streaming to an Apache Ignite cluster from embedded devices and real-time data processing with Apache Spark. + + + +https://www.meetup.com/Dublin-Spark-Meetup/events/243875410/"; target="_blank">Read more + + + + + +https://www.meetup.com/Dublin-Spark-Meetup/events/243875410/"; + data-a2a-title="Apache Spark and Apache Ignite: Where Fast Data Meets the IoT"> + + + +https://www.addtoany.com/share";> + + + + + How to Share State Across Multiple Spark Jobs using Apache® Ignite⢠Added: ignite/site/trunk/images/Ignite_tm_Logo_black_RGB.svg URL: http://svn.apache.org/viewvc/ignite/site/trunk/images/Ignite_tm_Logo_black_RGB.svg?rev=1812781&view=auto == --- ignite/site/trunk/images/Ignite_tm_Logo_black_RGB.svg (added) +++ ignite/site/trunk/images/Ignite_tm_Logo_black_RGB.svg Fri Oct 20 19:01:17 2017 @@ -0,0 +1 @@ +http://www.w3.org/2000/svg"; viewBox="0 0 582.76 265.7">.cls-1{fill:#ed1c24;}Igniteâ¢_Logo_Full_RGB \ No newline at end of file Added: ignite/site/trunk/images/Ignite_tm_Logo_white_RGB.svg URL: http://svn.apache.org/viewvc/ignite/site/trunk/images/Ignite_tm_Logo_white_RGB.svg?rev=1812781&view=auto == --- ignite/site/trunk/images/Ignite_tm_Logo_white_RGB.svg (added) +++ ignite/site/trunk/images/Ignite_tm_Logo_white_RGB.svg Fri Oct 20 19:01:17 2017 @@ -0,0 +1 @@ +http://www.w3.org/2000/svg"; viewBox="0 0 582.76 265.7">.cls-1{fill:#fff;}.cls-2{fill:#ed1c24;}Igniteâ¢_Logo_Inv_RGB \ No newline at end of file Modified: ignite/site/trunk/includes/footer.html URL: http://svn.apache.org/viewvc/ignite/site/trunk/includes/footer.html?rev=1812781&r1=1812780&r2=1812781&view=diff == --- ignite/site/trunk/includes/footer.html (original) +++ ignite/site/trunk/includes/footer.html Fri Oct 20 19:01:17 2017 @@ -3,7 +3,7 @@ - + https://donate.apache.org/";> Modified: ignite/site/trunk/includes/header.html URL: http://svn.apache.org/viewvc/ignite/site/trunk/includes/header.html?rev=1812781&r1=1812780&r2=1812781&view=diff == --- ignite/site/trunk/includes/header.html (original) +++ ignite/site/trunk/includes/header.html Fri Oct 20 19:01:17 2017 @@ -34,7 +34,7 @@ - + Modified: ignite/site/trunk/index.html URL: http://svn.apache.org/viewvc/ignite/site/trunk/index.html?rev=1812781&r1=1812780&r2=1812781&view=diff == Binary files - no diff available.
svn commit: r1812777 - in /ignite/site/trunk: features.html features/streaming.html
Author: dmagda Date: Fri Oct 20 18:53:11 2017 New Revision: 1812777 URL: http://svn.apache.org/viewvc?rev=1812777&view=rev Log: updated the streaming page Modified: ignite/site/trunk/features.html ignite/site/trunk/features/streaming.html Modified: ignite/site/trunk/features.html URL: http://svn.apache.org/viewvc/ignite/site/trunk/features.html?rev=1812777&r1=1812776&r2=1812777&view=diff == --- ignite/site/trunk/features.html (original) +++ ignite/site/trunk/features.html Fri Oct 20 18:53:11 2017 @@ -71,7 +71,7 @@ under the License. Extended Features Service Grid -Fast Data Ingest +Data Loading & Streaming RDBMS Integration Data Structures Messaging & Events Modified: ignite/site/trunk/features/streaming.html URL: http://svn.apache.org/viewvc/ignite/site/trunk/features/streaming.html?rev=1812777&r1=1812776&r2=1812777&view=diff == --- ignite/site/trunk/features/streaming.html (original) +++ ignite/site/trunk/features/streaming.html Fri Oct 20 18:53:11 2017 @@ -39,7 +39,7 @@ under the License. -Data Ingestion and Streaming - Apache Ignite +Data Loading and Streaming - Apache Ignite https://netdna.bootstrapcdn.com/font-awesome/4.2.0/css/font-awesome.css"; rel="stylesheet"> @@ -52,23 +52,27 @@ under the License. -Data Ingestion and Streaming +Data Loading and Streaming -Ignite streaming capabilities allows ingesting never-ending streams of data in a scalable -and fault-tolerant fashion. The rates at which data can be injected into Ignite can be very -high and easily exceed millions of events per second on a moderately sized cluster. +Ignite data loading and streaming capabilities allow ingesting large finite as well as +never-ending volumes of data in a scalable and fault-tolerant way into the cluster. +The rate at which data can be injected into Ignite is very high and easily exceeds millions +of events per second on a moderately sized cluster. -How it Works: - -Clients inject streams of data into Ignite. -Data is automatically partitioned between Ignite data nodes. -Data is concurrently processed across all cluster nodes. -Clients perform concurrent SQL queries on the streamed data. -Clients subscribe to continuous queries as data changes. - +Apache Ignite https://apacheignite-mix.readme.io/docs/overview"; target="_blank"> +integrates with major streaming technologies and frameworks such as Kafka, Camel, +Storm or JMS to bring even more advanced streaming capabilities to Ignite-based architectures. + + +Data Loading + +Ignite provides several https://apacheignite.readme.io/docs/data-loading"; target="_blank"> +techniques for initial data loading. For instance, Ignite streaming APIs are a good +choice for clusters with Ignite native persistence enabled, while the clusters that persist +data in a 3rd party store can connect to it directly with CacheStore API. @@ -78,6 +82,16 @@ under the License. + +How Ignite Streaming Works: + +Clients inject streams of data into Ignite. +Data is automatically partitioned between Ignite data nodes. +Data is concurrently processed across all cluster nodes. +Clients perform concurrent SQL queries on the streamed data. +Clients subscribe to continuous queries as data changes. + + Code Examples:
ignite git commit: AI release notes.
Repository: ignite Updated Branches: refs/heads/ignite-2.3 347696d24 -> e6cb5300d AI release notes. Project: http://git-wip-us.apache.org/repos/asf/ignite/repo Commit: http://git-wip-us.apache.org/repos/asf/ignite/commit/e6cb5300 Tree: http://git-wip-us.apache.org/repos/asf/ignite/tree/e6cb5300 Diff: http://git-wip-us.apache.org/repos/asf/ignite/diff/e6cb5300 Branch: refs/heads/ignite-2.3 Commit: e6cb5300d51c5184e876b988c4683bc605685874 Parents: 347696d Author: devozerov Authored: Fri Oct 20 18:11:52 2017 +0300 Committer: devozerov Committed: Fri Oct 20 18:12:46 2017 +0300 -- RELEASE_NOTES.txt | 97 ++ 1 file changed, 97 insertions(+) -- http://git-wip-us.apache.org/repos/asf/ignite/blob/e6cb5300/RELEASE_NOTES.txt -- diff --git a/RELEASE_NOTES.txt b/RELEASE_NOTES.txt index 23497d8..25007c7 100644 --- a/RELEASE_NOTES.txt +++ b/RELEASE_NOTES.txt @@ -1,6 +1,103 @@ Apache Ignite Release Notes === +Apache Ignite In-Memory Data Fabric 2.3 +--- +Ignite: +* Ability to enable persistence per data region. +* Default page size is changed to 4KB. +* Ability to enable and disable rebalancing per-node. +* Added methods for batch services deployment. +* Introduced cache start/stop order on cluster activation. +* EstimatedRebalancingFinishTime and RebalancingStartTime metrics are exposed through MXBean interface. +* Ability to (de)activate cluster via http-rest API. +* Added Ignite update notifier. +* Updated Lucene dependency to version 5.5.2. +* Added an option to disable MBeans. +* Added sorted and multithreaded checkpoint modes. +* Added calculation of fill factor memory metric in persistent mode. +* Added estimated time for rebalance start and completion to cache metrics. +* Added API to add binary metadata locally. +* Added write throttling during an ongoing checkpoint to avoid zero performance drops. +* Ability to convert WAL to human-redable form. +* Ability to handle CacheObject from DataRecord in standalone WAL iterator. +* Support for uninterruptable writes using AsyncFileIOFactory; enabled with -DIGNITE_USE_ASYNC_FILE_IO_FACTORY=true. +* Enhanced persistent store path resolving to not rely on automatically generated consistent ID. +* Pre-configure local event listeners with IgniteConfiguration.LocalEventListeners. +* Massive performance and stability improvements. + +Ignite.NET: +* LINQ: Local collections joins support. +* LINQ: Regex support. + +Ignite CPP: +* Added Compute::Broadcast method. +* Added Ignite::SetActive method. + +SQL: +* Added sqlline utility to the build. +* CREATE TABLE: Added NOT NULL support. +* CREATE TABLE: Ability to specify cache, key type and value type names. +* CREATE TABLE: Added "WRAP_KEY" and "WRAP_VALUE" options to CREATE TABLE command. +* CREATE TABLE: Added WRITE_SYNCHRONIZATION_MODE option. +* ALTER TABLE: ADD COLUMN support. +* Added lazy query execution mode (SqlFieldsQuery.setLazy). +* Added QuerySqlField.inlineSize property. +* Added FieldsQueryCursor interface to get fields metadata for SqlFieldsQuery. +* Added QueryEntity(Class keyClass, Class valueClass) constructor. +* Improved LocalDate, LocalTime and LocalDateTime support for Java 8. +* Optimized memory consumption during query execution. +* Fixed BigInteger data type handling. + +JDBC Driver: +* Batch update support. +* SQLSTATE error codes support. +* Added "enforceJoinOrder" flag to connection string. +* Added "skipReducerOnUpdate" flag. +* Thin driver: Allowed execution of multiple SQL statements in one command. +* Thin driver: Added metadata support. +* Thin driver: Added type conversions in accordance with specification. +* Thin driver: Added schema to connection string. +* Thin driver: Added Statement.closeOnCompletion() support. +* Appended UUID to thick driver's Ignite instance name to avoid collision with user-created instances. +* Fixed a bug in PreparedStatement not clearing query parameters after execution. + +ODBC Driver: +* SQLSTATE error codes support. +* Support for BINARY and VARBINARY types. +* DML statement batching. +* SQL_ATTR_CONNECTION_DEAD connection attribute. +* SQLGetInfo for info types. +* Added "skipReducerOnUpdate" flag. +* SQLRowCount now returns number of affected rows for non-batch queries. +* SQLBindParameter do not unbind parameter if the ParameterValuePtr is NULL anymore. +* Fixed error on cursor closing before all the rows from the result fetched. + +Machine Learning: +* Implemented K-means clustering algorithm optimized for distributed storages. +* Introduced sparse block distributed matrix. +* Initial BLAS implementation. +* Integration with BLAS via netlib. +* Added getRow() and getCol() methods to Matrix API. + +Web Console: +* Improved DDL s
ignite git commit: AI release notes.
Repository: ignite Updated Branches: refs/heads/master 280a22ab7 -> 3b18170ff AI release notes. Project: http://git-wip-us.apache.org/repos/asf/ignite/repo Commit: http://git-wip-us.apache.org/repos/asf/ignite/commit/3b18170f Tree: http://git-wip-us.apache.org/repos/asf/ignite/tree/3b18170f Diff: http://git-wip-us.apache.org/repos/asf/ignite/diff/3b18170f Branch: refs/heads/master Commit: 3b18170ffc80061278e3ed6ffc3296aed4644161 Parents: 280a22a Author: devozerov Authored: Fri Oct 20 18:11:52 2017 +0300 Committer: devozerov Committed: Fri Oct 20 18:11:52 2017 +0300 -- RELEASE_NOTES.txt | 97 ++ 1 file changed, 97 insertions(+) -- http://git-wip-us.apache.org/repos/asf/ignite/blob/3b18170f/RELEASE_NOTES.txt -- diff --git a/RELEASE_NOTES.txt b/RELEASE_NOTES.txt index 23497d8..25007c7 100644 --- a/RELEASE_NOTES.txt +++ b/RELEASE_NOTES.txt @@ -1,6 +1,103 @@ Apache Ignite Release Notes === +Apache Ignite In-Memory Data Fabric 2.3 +--- +Ignite: +* Ability to enable persistence per data region. +* Default page size is changed to 4KB. +* Ability to enable and disable rebalancing per-node. +* Added methods for batch services deployment. +* Introduced cache start/stop order on cluster activation. +* EstimatedRebalancingFinishTime and RebalancingStartTime metrics are exposed through MXBean interface. +* Ability to (de)activate cluster via http-rest API. +* Added Ignite update notifier. +* Updated Lucene dependency to version 5.5.2. +* Added an option to disable MBeans. +* Added sorted and multithreaded checkpoint modes. +* Added calculation of fill factor memory metric in persistent mode. +* Added estimated time for rebalance start and completion to cache metrics. +* Added API to add binary metadata locally. +* Added write throttling during an ongoing checkpoint to avoid zero performance drops. +* Ability to convert WAL to human-redable form. +* Ability to handle CacheObject from DataRecord in standalone WAL iterator. +* Support for uninterruptable writes using AsyncFileIOFactory; enabled with -DIGNITE_USE_ASYNC_FILE_IO_FACTORY=true. +* Enhanced persistent store path resolving to not rely on automatically generated consistent ID. +* Pre-configure local event listeners with IgniteConfiguration.LocalEventListeners. +* Massive performance and stability improvements. + +Ignite.NET: +* LINQ: Local collections joins support. +* LINQ: Regex support. + +Ignite CPP: +* Added Compute::Broadcast method. +* Added Ignite::SetActive method. + +SQL: +* Added sqlline utility to the build. +* CREATE TABLE: Added NOT NULL support. +* CREATE TABLE: Ability to specify cache, key type and value type names. +* CREATE TABLE: Added "WRAP_KEY" and "WRAP_VALUE" options to CREATE TABLE command. +* CREATE TABLE: Added WRITE_SYNCHRONIZATION_MODE option. +* ALTER TABLE: ADD COLUMN support. +* Added lazy query execution mode (SqlFieldsQuery.setLazy). +* Added QuerySqlField.inlineSize property. +* Added FieldsQueryCursor interface to get fields metadata for SqlFieldsQuery. +* Added QueryEntity(Class keyClass, Class valueClass) constructor. +* Improved LocalDate, LocalTime and LocalDateTime support for Java 8. +* Optimized memory consumption during query execution. +* Fixed BigInteger data type handling. + +JDBC Driver: +* Batch update support. +* SQLSTATE error codes support. +* Added "enforceJoinOrder" flag to connection string. +* Added "skipReducerOnUpdate" flag. +* Thin driver: Allowed execution of multiple SQL statements in one command. +* Thin driver: Added metadata support. +* Thin driver: Added type conversions in accordance with specification. +* Thin driver: Added schema to connection string. +* Thin driver: Added Statement.closeOnCompletion() support. +* Appended UUID to thick driver's Ignite instance name to avoid collision with user-created instances. +* Fixed a bug in PreparedStatement not clearing query parameters after execution. + +ODBC Driver: +* SQLSTATE error codes support. +* Support for BINARY and VARBINARY types. +* DML statement batching. +* SQL_ATTR_CONNECTION_DEAD connection attribute. +* SQLGetInfo for info types. +* Added "skipReducerOnUpdate" flag. +* SQLRowCount now returns number of affected rows for non-batch queries. +* SQLBindParameter do not unbind parameter if the ParameterValuePtr is NULL anymore. +* Fixed error on cursor closing before all the rows from the result fetched. + +Machine Learning: +* Implemented K-means clustering algorithm optimized for distributed storages. +* Introduced sparse block distributed matrix. +* Initial BLAS implementation. +* Integration with BLAS via netlib. +* Added getRow() and getCol() methods to Matrix API. + +Web Console: +* Improved DDL support.
ignite git commit: Removed redundant libs from libs/optional
Repository: ignite Updated Branches: refs/heads/master 370071738 -> 280a22ab7 Removed redundant libs from libs/optional (cherry picked from commit 347696d) Project: http://git-wip-us.apache.org/repos/asf/ignite/repo Commit: http://git-wip-us.apache.org/repos/asf/ignite/commit/280a22ab Tree: http://git-wip-us.apache.org/repos/asf/ignite/tree/280a22ab Diff: http://git-wip-us.apache.org/repos/asf/ignite/diff/280a22ab Branch: refs/heads/master Commit: 280a22ab759e2d4395e30dbdf04bedce2f6c8c11 Parents: 3700717 Author: Anton Vinogradov Authored: Fri Oct 20 17:15:39 2017 +0300 Committer: Anton Vinogradov Committed: Fri Oct 20 17:17:27 2017 +0300 -- assembly/dependencies-fabric-lgpl.xml | 2 ++ assembly/dependencies-fabric.xml | 2 ++ 2 files changed, 4 insertions(+) -- http://git-wip-us.apache.org/repos/asf/ignite/blob/280a22ab/assembly/dependencies-fabric-lgpl.xml -- diff --git a/assembly/dependencies-fabric-lgpl.xml b/assembly/dependencies-fabric-lgpl.xml index e2fac3c..82b5d5c 100644 --- a/assembly/dependencies-fabric-lgpl.xml +++ b/assembly/dependencies-fabric-lgpl.xml @@ -133,6 +133,8 @@ org.apache.ignite:ignite-yardstick org.apache.ignite:ignite-benchmarks org.apache.ignite:ignite-web-agent +org.apache.ignite:ignite-dev-utils +org.apache.ignite:ignite-extdata-platform true http://git-wip-us.apache.org/repos/asf/ignite/blob/280a22ab/assembly/dependencies-fabric.xml -- diff --git a/assembly/dependencies-fabric.xml b/assembly/dependencies-fabric.xml index 6c4101e..3e9405b 100644 --- a/assembly/dependencies-fabric.xml +++ b/assembly/dependencies-fabric.xml @@ -138,6 +138,8 @@ org.apache.ignite:ignite-yardstick org.apache.ignite:ignite-benchmarks org.apache.ignite:ignite-web-agent +org.apache.ignite:ignite-dev-utils +org.apache.ignite:ignite-extdata-platform true
ignite git commit: Removed redundant libs from libs/optional
Repository: ignite Updated Branches: refs/heads/ignite-2.3 8ee033fdc -> 347696d24 Removed redundant libs from libs/optional Project: http://git-wip-us.apache.org/repos/asf/ignite/repo Commit: http://git-wip-us.apache.org/repos/asf/ignite/commit/347696d2 Tree: http://git-wip-us.apache.org/repos/asf/ignite/tree/347696d2 Diff: http://git-wip-us.apache.org/repos/asf/ignite/diff/347696d2 Branch: refs/heads/ignite-2.3 Commit: 347696d2426ef5be8294253141e299097a6564cc Parents: 8ee033f Author: Anton Vinogradov Authored: Fri Oct 20 17:15:39 2017 +0300 Committer: Anton Vinogradov Committed: Fri Oct 20 17:15:39 2017 +0300 -- assembly/dependencies-fabric-lgpl.xml | 2 ++ assembly/dependencies-fabric.xml | 2 ++ 2 files changed, 4 insertions(+) -- http://git-wip-us.apache.org/repos/asf/ignite/blob/347696d2/assembly/dependencies-fabric-lgpl.xml -- diff --git a/assembly/dependencies-fabric-lgpl.xml b/assembly/dependencies-fabric-lgpl.xml index e2fac3c..82b5d5c 100644 --- a/assembly/dependencies-fabric-lgpl.xml +++ b/assembly/dependencies-fabric-lgpl.xml @@ -133,6 +133,8 @@ org.apache.ignite:ignite-yardstick org.apache.ignite:ignite-benchmarks org.apache.ignite:ignite-web-agent +org.apache.ignite:ignite-dev-utils +org.apache.ignite:ignite-extdata-platform true http://git-wip-us.apache.org/repos/asf/ignite/blob/347696d2/assembly/dependencies-fabric.xml -- diff --git a/assembly/dependencies-fabric.xml b/assembly/dependencies-fabric.xml index 6c4101e..3e9405b 100644 --- a/assembly/dependencies-fabric.xml +++ b/assembly/dependencies-fabric.xml @@ -138,6 +138,8 @@ org.apache.ignite:ignite-yardstick org.apache.ignite:ignite-benchmarks org.apache.ignite:ignite-web-agent +org.apache.ignite:ignite-dev-utils +org.apache.ignite:ignite-extdata-platform true
ignite git commit: IGNITE-6647 Web Console: Added folder for modules migrations.
Repository: ignite Updated Branches: refs/heads/master ab08be83c -> 370071738 IGNITE-6647 Web Console: Added folder for modules migrations. Project: http://git-wip-us.apache.org/repos/asf/ignite/repo Commit: http://git-wip-us.apache.org/repos/asf/ignite/commit/37007173 Tree: http://git-wip-us.apache.org/repos/asf/ignite/tree/37007173 Diff: http://git-wip-us.apache.org/repos/asf/ignite/diff/37007173 Branch: refs/heads/master Commit: 370071738baad3f568c17fb6145ad788ed1d03c2 Parents: ab08be8 Author: Alexey Kuznetsov Authored: Fri Oct 20 21:15:02 2017 +0700 Committer: Alexey Kuznetsov Committed: Fri Oct 20 21:15:02 2017 +0700 -- modules/web-console/backend/ignite_modules/migrations/README.txt | 4 1 file changed, 4 insertions(+) -- http://git-wip-us.apache.org/repos/asf/ignite/blob/37007173/modules/web-console/backend/ignite_modules/migrations/README.txt -- diff --git a/modules/web-console/backend/ignite_modules/migrations/README.txt b/modules/web-console/backend/ignite_modules/migrations/README.txt new file mode 100644 index 000..daeae36 --- /dev/null +++ b/modules/web-console/backend/ignite_modules/migrations/README.txt @@ -0,0 +1,4 @@ +Ignite Web Console +== + +This folder contains scripts for modules model migration.
[1/3] ignite git commit: IGNITE-6515 .NET: Enable persistence on per-cache basis
Repository: ignite Updated Branches: refs/heads/ignite-2.3 6c58b4ac7 -> 8ee033fdc http://git-wip-us.apache.org/repos/asf/ignite/blob/8ee033fd/modules/platforms/dotnet/Apache.Ignite.Core/IgniteConfigurationSection.xsd -- diff --git a/modules/platforms/dotnet/Apache.Ignite.Core/IgniteConfigurationSection.xsd b/modules/platforms/dotnet/Apache.Ignite.Core/IgniteConfigurationSection.xsd index 988fa1f..6ede267 100644 --- a/modules/platforms/dotnet/Apache.Ignite.Core/IgniteConfigurationSection.xsd +++ b/modules/platforms/dotnet/Apache.Ignite.Core/IgniteConfigurationSection.xsd @@ -759,9 +759,9 @@ Value indicating whether statistics gathering is enabled on a cache. These statistics can be retrieved via ICache.GetMetrics(). - + -Name of the MemoryPolicyConfiguration for this cache. +Name of the DataRegionConfiguration for this cache. @@ -1380,7 +1380,7 @@ -Persistent store configuration. +Persistent store configuration. Obsolete, use DataStorageConfiguration. @@ -1490,6 +1490,273 @@ + + +Data storage configuration. + + + + + +Default data region configuration. + + + + +Minimal number of empty pages to be present in reuse lists for this memory policy. + + + + +Threshold for memory pages eviction initiation. For instance, if the threshold is 0.9 it means that the page memory will start the eviction only after 90% of the memory region (defined by this policy) is occupied. + + + + +Data region name. + + + + +Page eviction mode. + + + + +Initial data region size. + + + + +Maximum data region size. + + + + +Path to the directory for memory-mapped files. + + + + +Enable memory metrics. + + + + +Number of sub intervals to split RateTimeInterval into. + + + + +Rate time interval for AllocationRate and EvictionRate monitoring. + + + + +Enable disk persistence for this region. + +
[3/3] ignite git commit: IGNITE-6515 .NET: Enable persistence on per-cache basis
IGNITE-6515 .NET: Enable persistence on per-cache basis This closes #2891 Project: http://git-wip-us.apache.org/repos/asf/ignite/repo Commit: http://git-wip-us.apache.org/repos/asf/ignite/commit/8ee033fd Tree: http://git-wip-us.apache.org/repos/asf/ignite/tree/8ee033fd Diff: http://git-wip-us.apache.org/repos/asf/ignite/diff/8ee033fd Branch: refs/heads/ignite-2.3 Commit: 8ee033fdc50b11c6913e1b6ddc100c28f6bf4341 Parents: 6c58b4a Author: Pavel Tupitsyn Authored: Fri Oct 20 14:38:11 2017 +0300 Committer: Pavel Tupitsyn Committed: Fri Oct 20 14:40:56 2017 +0300 -- .../platform/cluster/PlatformClusterGroup.java | 90 .../utils/PlatformConfigurationUtils.java | 174 ++- .../Apache.Ignite.Core.Tests.csproj | 5 +- .../Cache/CacheConfigurationTest.cs | 6 + .../Cache/DataRegionMetricsTest.cs | 153 ++ .../Cache/DataStorageMetricsTest.cs | 107 + .../Cache/MemoryMetricsTest.cs | 1 + .../Cache/PersistenceTest.cs| 235 ++ .../Cache/PersistentStoreTest.cs| 189 .../Cache/PersistentStoreTestObsolete.cs| 190 .../Config/full-config.xml | 18 + .../Config/spring-test.xml | 18 +- .../IgniteConfigurationSerializerTest.cs| 135 +- .../IgniteConfigurationTest.cs | 227 + .../Apache.Ignite.Core.csproj | 10 + .../Cache/Configuration/CacheConfiguration.cs | 21 +- .../Cache/Configuration/DataPageEvictionMode.cs | 3 + .../Cache/Configuration/MemoryConfiguration.cs | 5 + .../Configuration/MemoryPolicyConfiguration.cs | 3 + .../Apache.Ignite.Core/Cache/IMemoryMetrics.cs | 4 + .../Configuration/CheckpointWriteOrder.cs | 37 ++ .../Configuration/DataPageEvictionMode.cs | 59 +++ .../Configuration/DataRegionConfiguration.cs| 213 + .../Configuration/DataStorageConfiguration.cs | 466 +++ .../Apache.Ignite.Core/Configuration/WalMode.cs | 45 ++ .../Apache.Ignite.Core/IDataRegionMetrics.cs| 55 +++ .../Apache.Ignite.Core/IDataStorageMetrics.cs | 87 .../dotnet/Apache.Ignite.Core/IIgnite.cs| 33 ++ .../Apache.Ignite.Core/IgniteConfiguration.cs | 40 +- .../IgniteConfigurationSection.xsd | 273 ++- .../Impl/Cache/MemoryMetrics.cs | 2 + .../Impl/Cluster/ClusterGroupImpl.cs| 53 +++ .../Common/IgniteConfigurationXmlSerializer.cs | 11 +- .../Impl/DataRegionMetrics.cs | 61 +++ .../Impl/DataStorageMetrics.cs | 87 .../dotnet/Apache.Ignite.Core/Impl/Ignite.cs| 22 + .../PersistentStore/PersistentStoreMetrics.cs | 2 + .../PersistentStore/CheckpointWriteOrder.cs | 3 + .../PersistentStore/IPersistentStoreMetrics.cs | 2 + .../PersistentStoreConfiguration.cs | 4 + .../PersistentStore/WalMode.cs | 3 + 41 files changed, 2801 insertions(+), 351 deletions(-) -- http://git-wip-us.apache.org/repos/asf/ignite/blob/8ee033fd/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/cluster/PlatformClusterGroup.java -- diff --git a/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/cluster/PlatformClusterGroup.java b/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/cluster/PlatformClusterGroup.java index 7c1c03e..ef382d6 100644 --- a/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/cluster/PlatformClusterGroup.java +++ b/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/cluster/PlatformClusterGroup.java @@ -21,6 +21,8 @@ import java.util.ArrayList; import java.util.Collection; import java.util.UUID; +import org.apache.ignite.DataRegionMetrics; +import org.apache.ignite.DataStorageMetrics; import org.apache.ignite.IgniteCache; import org.apache.ignite.IgniteCheckedException; import org.apache.ignite.IgniteCluster; @@ -142,6 +144,14 @@ public class PlatformClusterGroup extends PlatformAbstractTarget { /** */ private static final int OP_GET_SERVICES = 34; +/** */ +private static final int OP_DATA_REGION_METRICS = 35; + +/** */ +private static final int OP_DATA_REGION_METRICS_BY_NAME = 36; + +/** */ +private static final int OP_DATA_STORAGE_METRICS = 37; /** Projection. */ private final ClusterGroupEx prj; @@ -187,6 +197,26 @@ public class PlatformClusterGroup extends PlatformAbstractTarget { break; } +case OP_DATA_STORAGE_METRICS: { +DataStorageMetrics metrics = prj.ignite().dataStorageMetrics(); + +wr
[2/3] ignite git commit: IGNITE-6515 .NET: Enable persistence on per-cache basis
http://git-wip-us.apache.org/repos/asf/ignite/blob/8ee033fd/modules/platforms/dotnet/Apache.Ignite.Core.Tests/IgniteConfigurationTest.cs -- diff --git a/modules/platforms/dotnet/Apache.Ignite.Core.Tests/IgniteConfigurationTest.cs b/modules/platforms/dotnet/Apache.Ignite.Core.Tests/IgniteConfigurationTest.cs index 73636d1..c8c06b2 100644 --- a/modules/platforms/dotnet/Apache.Ignite.Core.Tests/IgniteConfigurationTest.cs +++ b/modules/platforms/dotnet/Apache.Ignite.Core.Tests/IgniteConfigurationTest.cs @@ -41,6 +41,7 @@ namespace Apache.Ignite.Core.Tests using Apache.Ignite.Core.Tests.Plugin; using Apache.Ignite.Core.Transactions; using NUnit.Framework; +using WalMode = Apache.Ignite.Core.PersistentStore.WalMode; /// /// Tests code-based configuration. @@ -64,6 +65,8 @@ namespace Apache.Ignite.Core.Tests { CheckDefaultProperties(new IgniteConfiguration()); CheckDefaultProperties(new PersistentStoreConfiguration()); +CheckDefaultProperties(new DataStorageConfiguration()); +CheckDefaultProperties(new DataRegionConfiguration()); CheckDefaultProperties(new ClientConnectorConfiguration()); CheckDefaultProperties(new SqlConnectorConfiguration()); } @@ -94,6 +97,8 @@ namespace Apache.Ignite.Core.Tests CheckDefaultValueAttributes(new PersistentStoreConfiguration()); CheckDefaultValueAttributes(new IgniteClientConfiguration()); CheckDefaultValueAttributes(new QueryIndex()); +CheckDefaultValueAttributes(new DataStorageConfiguration()); +CheckDefaultValueAttributes(new DataRegionConfiguration()); } /// @@ -219,33 +224,6 @@ namespace Apache.Ignite.Core.Tests Assert.AreEqual(eventCfg.ExpirationTimeout, resEventCfg.ExpirationTimeout); Assert.AreEqual(eventCfg.MaxEventCount, resEventCfg.MaxEventCount); -var memCfg = cfg.MemoryConfiguration; -var resMemCfg = resCfg.MemoryConfiguration; -Assert.IsNotNull(memCfg); -Assert.IsNotNull(resMemCfg); -Assert.AreEqual(memCfg.PageSize, resMemCfg.PageSize); -Assert.AreEqual(memCfg.ConcurrencyLevel, resMemCfg.ConcurrencyLevel); -Assert.AreEqual(memCfg.DefaultMemoryPolicyName, resMemCfg.DefaultMemoryPolicyName); -Assert.AreEqual(memCfg.SystemCacheInitialSize, resMemCfg.SystemCacheInitialSize); -Assert.AreEqual(memCfg.SystemCacheMaxSize, resMemCfg.SystemCacheMaxSize); -Assert.IsNotNull(memCfg.MemoryPolicies); -Assert.IsNotNull(resMemCfg.MemoryPolicies); -Assert.AreEqual(2, memCfg.MemoryPolicies.Count); -Assert.AreEqual(2, resMemCfg.MemoryPolicies.Count); - -for (var i = 0; i < memCfg.MemoryPolicies.Count; i++) -{ -var plc = memCfg.MemoryPolicies.Skip(i).First(); -var resPlc = resMemCfg.MemoryPolicies.Skip(i).First(); - -Assert.AreEqual(plc.PageEvictionMode, resPlc.PageEvictionMode); -Assert.AreEqual(plc.MaxSize, resPlc.MaxSize); -Assert.AreEqual(plc.EmptyPagesPoolSize, resPlc.EmptyPagesPoolSize); -Assert.AreEqual(plc.EvictionThreshold, resPlc.EvictionThreshold); -Assert.AreEqual(plc.Name, resPlc.Name); -Assert.AreEqual(plc.SwapFilePath, resPlc.SwapFilePath); -} - var sql = cfg.SqlConnectorConfiguration; var resSql = resCfg.SqlConnectorConfiguration; @@ -258,30 +236,7 @@ namespace Apache.Ignite.Core.Tests Assert.AreEqual(sql.TcpNoDelay, resSql.TcpNoDelay); Assert.AreEqual(sql.ThreadPoolSize, resSql.ThreadPoolSize); -var pers = cfg.PersistentStoreConfiguration; -var resPers = resCfg.PersistentStoreConfiguration; - -Assert.AreEqual(pers.AlwaysWriteFullPages, resPers.AlwaysWriteFullPages); -Assert.AreEqual(pers.CheckpointingFrequency, resPers.CheckpointingFrequency); -Assert.AreEqual(pers.CheckpointingPageBufferSize, resPers.CheckpointingPageBufferSize); -Assert.AreEqual(pers.CheckpointingThreads, resPers.CheckpointingThreads); -Assert.AreEqual(pers.LockWaitTime, resPers.LockWaitTime); -Assert.AreEqual(pers.PersistentStorePath, resPers.PersistentStorePath); -Assert.AreEqual(pers.TlbSize, resPers.TlbSize); -Assert.AreEqual(pers.WalArchivePath, resPers.WalArchivePath); -Assert.AreEqual(pers.WalFlushFrequency, resPers.WalFlushFrequency); -Assert.AreEqual(pers.WalFsyncDelayNanos, resPers.WalFsyncDelayNa
[2/3] ignite git commit: IGNITE-6515 .NET: Enable persistence on per-cache basis
http://git-wip-us.apache.org/repos/asf/ignite/blob/ab08be83/modules/platforms/dotnet/Apache.Ignite.Core.Tests/IgniteConfigurationTest.cs -- diff --git a/modules/platforms/dotnet/Apache.Ignite.Core.Tests/IgniteConfigurationTest.cs b/modules/platforms/dotnet/Apache.Ignite.Core.Tests/IgniteConfigurationTest.cs index 73636d1..c8c06b2 100644 --- a/modules/platforms/dotnet/Apache.Ignite.Core.Tests/IgniteConfigurationTest.cs +++ b/modules/platforms/dotnet/Apache.Ignite.Core.Tests/IgniteConfigurationTest.cs @@ -41,6 +41,7 @@ namespace Apache.Ignite.Core.Tests using Apache.Ignite.Core.Tests.Plugin; using Apache.Ignite.Core.Transactions; using NUnit.Framework; +using WalMode = Apache.Ignite.Core.PersistentStore.WalMode; /// /// Tests code-based configuration. @@ -64,6 +65,8 @@ namespace Apache.Ignite.Core.Tests { CheckDefaultProperties(new IgniteConfiguration()); CheckDefaultProperties(new PersistentStoreConfiguration()); +CheckDefaultProperties(new DataStorageConfiguration()); +CheckDefaultProperties(new DataRegionConfiguration()); CheckDefaultProperties(new ClientConnectorConfiguration()); CheckDefaultProperties(new SqlConnectorConfiguration()); } @@ -94,6 +97,8 @@ namespace Apache.Ignite.Core.Tests CheckDefaultValueAttributes(new PersistentStoreConfiguration()); CheckDefaultValueAttributes(new IgniteClientConfiguration()); CheckDefaultValueAttributes(new QueryIndex()); +CheckDefaultValueAttributes(new DataStorageConfiguration()); +CheckDefaultValueAttributes(new DataRegionConfiguration()); } /// @@ -219,33 +224,6 @@ namespace Apache.Ignite.Core.Tests Assert.AreEqual(eventCfg.ExpirationTimeout, resEventCfg.ExpirationTimeout); Assert.AreEqual(eventCfg.MaxEventCount, resEventCfg.MaxEventCount); -var memCfg = cfg.MemoryConfiguration; -var resMemCfg = resCfg.MemoryConfiguration; -Assert.IsNotNull(memCfg); -Assert.IsNotNull(resMemCfg); -Assert.AreEqual(memCfg.PageSize, resMemCfg.PageSize); -Assert.AreEqual(memCfg.ConcurrencyLevel, resMemCfg.ConcurrencyLevel); -Assert.AreEqual(memCfg.DefaultMemoryPolicyName, resMemCfg.DefaultMemoryPolicyName); -Assert.AreEqual(memCfg.SystemCacheInitialSize, resMemCfg.SystemCacheInitialSize); -Assert.AreEqual(memCfg.SystemCacheMaxSize, resMemCfg.SystemCacheMaxSize); -Assert.IsNotNull(memCfg.MemoryPolicies); -Assert.IsNotNull(resMemCfg.MemoryPolicies); -Assert.AreEqual(2, memCfg.MemoryPolicies.Count); -Assert.AreEqual(2, resMemCfg.MemoryPolicies.Count); - -for (var i = 0; i < memCfg.MemoryPolicies.Count; i++) -{ -var plc = memCfg.MemoryPolicies.Skip(i).First(); -var resPlc = resMemCfg.MemoryPolicies.Skip(i).First(); - -Assert.AreEqual(plc.PageEvictionMode, resPlc.PageEvictionMode); -Assert.AreEqual(plc.MaxSize, resPlc.MaxSize); -Assert.AreEqual(plc.EmptyPagesPoolSize, resPlc.EmptyPagesPoolSize); -Assert.AreEqual(plc.EvictionThreshold, resPlc.EvictionThreshold); -Assert.AreEqual(plc.Name, resPlc.Name); -Assert.AreEqual(plc.SwapFilePath, resPlc.SwapFilePath); -} - var sql = cfg.SqlConnectorConfiguration; var resSql = resCfg.SqlConnectorConfiguration; @@ -258,30 +236,7 @@ namespace Apache.Ignite.Core.Tests Assert.AreEqual(sql.TcpNoDelay, resSql.TcpNoDelay); Assert.AreEqual(sql.ThreadPoolSize, resSql.ThreadPoolSize); -var pers = cfg.PersistentStoreConfiguration; -var resPers = resCfg.PersistentStoreConfiguration; - -Assert.AreEqual(pers.AlwaysWriteFullPages, resPers.AlwaysWriteFullPages); -Assert.AreEqual(pers.CheckpointingFrequency, resPers.CheckpointingFrequency); -Assert.AreEqual(pers.CheckpointingPageBufferSize, resPers.CheckpointingPageBufferSize); -Assert.AreEqual(pers.CheckpointingThreads, resPers.CheckpointingThreads); -Assert.AreEqual(pers.LockWaitTime, resPers.LockWaitTime); -Assert.AreEqual(pers.PersistentStorePath, resPers.PersistentStorePath); -Assert.AreEqual(pers.TlbSize, resPers.TlbSize); -Assert.AreEqual(pers.WalArchivePath, resPers.WalArchivePath); -Assert.AreEqual(pers.WalFlushFrequency, resPers.WalFlushFrequency); -Assert.AreEqual(pers.WalFsyncDelayNanos, resPers.WalFsyncDelayNa
[3/3] ignite git commit: IGNITE-6515 .NET: Enable persistence on per-cache basis
IGNITE-6515 .NET: Enable persistence on per-cache basis This closes #2891 Project: http://git-wip-us.apache.org/repos/asf/ignite/repo Commit: http://git-wip-us.apache.org/repos/asf/ignite/commit/ab08be83 Tree: http://git-wip-us.apache.org/repos/asf/ignite/tree/ab08be83 Diff: http://git-wip-us.apache.org/repos/asf/ignite/diff/ab08be83 Branch: refs/heads/master Commit: ab08be83ccc6fe1997ef80c25c3b48d45f410b56 Parents: ec9a945 Author: Pavel Tupitsyn Authored: Fri Oct 20 14:38:11 2017 +0300 Committer: Pavel Tupitsyn Committed: Fri Oct 20 14:38:11 2017 +0300 -- .../platform/cluster/PlatformClusterGroup.java | 90 .../utils/PlatformConfigurationUtils.java | 174 ++- .../Apache.Ignite.Core.Tests.csproj | 5 +- .../Cache/CacheConfigurationTest.cs | 6 + .../Cache/DataRegionMetricsTest.cs | 153 ++ .../Cache/DataStorageMetricsTest.cs | 107 + .../Cache/MemoryMetricsTest.cs | 1 + .../Cache/PersistenceTest.cs| 235 ++ .../Cache/PersistentStoreTest.cs| 189 .../Cache/PersistentStoreTestObsolete.cs| 190 .../Config/full-config.xml | 18 + .../Config/spring-test.xml | 18 +- .../IgniteConfigurationSerializerTest.cs| 135 +- .../IgniteConfigurationTest.cs | 227 + .../Apache.Ignite.Core.csproj | 10 + .../Cache/Configuration/CacheConfiguration.cs | 21 +- .../Cache/Configuration/DataPageEvictionMode.cs | 3 + .../Cache/Configuration/MemoryConfiguration.cs | 5 + .../Configuration/MemoryPolicyConfiguration.cs | 3 + .../Apache.Ignite.Core/Cache/IMemoryMetrics.cs | 4 + .../Configuration/CheckpointWriteOrder.cs | 37 ++ .../Configuration/DataPageEvictionMode.cs | 59 +++ .../Configuration/DataRegionConfiguration.cs| 213 + .../Configuration/DataStorageConfiguration.cs | 466 +++ .../Apache.Ignite.Core/Configuration/WalMode.cs | 45 ++ .../Apache.Ignite.Core/IDataRegionMetrics.cs| 55 +++ .../Apache.Ignite.Core/IDataStorageMetrics.cs | 87 .../dotnet/Apache.Ignite.Core/IIgnite.cs| 33 ++ .../Apache.Ignite.Core/IgniteConfiguration.cs | 40 +- .../IgniteConfigurationSection.xsd | 273 ++- .../Impl/Cache/MemoryMetrics.cs | 2 + .../Impl/Cluster/ClusterGroupImpl.cs| 53 +++ .../Common/IgniteConfigurationXmlSerializer.cs | 11 +- .../Impl/DataRegionMetrics.cs | 61 +++ .../Impl/DataStorageMetrics.cs | 87 .../dotnet/Apache.Ignite.Core/Impl/Ignite.cs| 22 + .../PersistentStore/PersistentStoreMetrics.cs | 2 + .../PersistentStore/CheckpointWriteOrder.cs | 3 + .../PersistentStore/IPersistentStoreMetrics.cs | 2 + .../PersistentStoreConfiguration.cs | 4 + .../PersistentStore/WalMode.cs | 3 + 41 files changed, 2801 insertions(+), 351 deletions(-) -- http://git-wip-us.apache.org/repos/asf/ignite/blob/ab08be83/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/cluster/PlatformClusterGroup.java -- diff --git a/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/cluster/PlatformClusterGroup.java b/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/cluster/PlatformClusterGroup.java index 7c1c03e..ef382d6 100644 --- a/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/cluster/PlatformClusterGroup.java +++ b/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/cluster/PlatformClusterGroup.java @@ -21,6 +21,8 @@ import java.util.ArrayList; import java.util.Collection; import java.util.UUID; +import org.apache.ignite.DataRegionMetrics; +import org.apache.ignite.DataStorageMetrics; import org.apache.ignite.IgniteCache; import org.apache.ignite.IgniteCheckedException; import org.apache.ignite.IgniteCluster; @@ -142,6 +144,14 @@ public class PlatformClusterGroup extends PlatformAbstractTarget { /** */ private static final int OP_GET_SERVICES = 34; +/** */ +private static final int OP_DATA_REGION_METRICS = 35; + +/** */ +private static final int OP_DATA_REGION_METRICS_BY_NAME = 36; + +/** */ +private static final int OP_DATA_STORAGE_METRICS = 37; /** Projection. */ private final ClusterGroupEx prj; @@ -187,6 +197,26 @@ public class PlatformClusterGroup extends PlatformAbstractTarget { break; } +case OP_DATA_STORAGE_METRICS: { +DataStorageMetrics metrics = prj.ignite().dataStorageMetrics(); + +writeD
[1/3] ignite git commit: IGNITE-6515 .NET: Enable persistence on per-cache basis
Repository: ignite Updated Branches: refs/heads/master ec9a945a1 -> ab08be83c http://git-wip-us.apache.org/repos/asf/ignite/blob/ab08be83/modules/platforms/dotnet/Apache.Ignite.Core/IgniteConfigurationSection.xsd -- diff --git a/modules/platforms/dotnet/Apache.Ignite.Core/IgniteConfigurationSection.xsd b/modules/platforms/dotnet/Apache.Ignite.Core/IgniteConfigurationSection.xsd index 988fa1f..6ede267 100644 --- a/modules/platforms/dotnet/Apache.Ignite.Core/IgniteConfigurationSection.xsd +++ b/modules/platforms/dotnet/Apache.Ignite.Core/IgniteConfigurationSection.xsd @@ -759,9 +759,9 @@ Value indicating whether statistics gathering is enabled on a cache. These statistics can be retrieved via ICache.GetMetrics(). - + -Name of the MemoryPolicyConfiguration for this cache. +Name of the DataRegionConfiguration for this cache. @@ -1380,7 +1380,7 @@ -Persistent store configuration. +Persistent store configuration. Obsolete, use DataStorageConfiguration. @@ -1490,6 +1490,273 @@ + + +Data storage configuration. + + + + + +Default data region configuration. + + + + +Minimal number of empty pages to be present in reuse lists for this memory policy. + + + + +Threshold for memory pages eviction initiation. For instance, if the threshold is 0.9 it means that the page memory will start the eviction only after 90% of the memory region (defined by this policy) is occupied. + + + + +Data region name. + + + + +Page eviction mode. + + + + +Initial data region size. + + + + +Maximum data region size. + + + + +Path to the directory for memory-mapped files. + + + + +Enable memory metrics. + + + + +Number of sub intervals to split RateTimeInterval into. + + + + +Rate time interval for AllocationRate and EvictionRate monitoring. + + + + +Enable disk persistence for this region. + +
ignite git commit: IGNITE-6030 Fixed misspelled metric
Repository: ignite Updated Branches: refs/heads/ignite-2.3 745677631 -> 6c58b4ac7 IGNITE-6030 Fixed misspelled metric Project: http://git-wip-us.apache.org/repos/asf/ignite/repo Commit: http://git-wip-us.apache.org/repos/asf/ignite/commit/6c58b4ac Tree: http://git-wip-us.apache.org/repos/asf/ignite/tree/6c58b4ac Diff: http://git-wip-us.apache.org/repos/asf/ignite/diff/6c58b4ac Branch: refs/heads/ignite-2.3 Commit: 6c58b4ac7c4527d583de49c4d8b250436273294c Parents: 7456776 Author: Alexey Goncharuk Authored: Fri Oct 20 13:43:44 2017 +0300 Committer: Alexey Goncharuk Committed: Fri Oct 20 13:44:48 2017 +0300 -- .../core/src/main/java/org/apache/ignite/DataStorageMetrics.java | 2 +- .../main/java/org/apache/ignite/DataStorageMetricsAdapter.java | 2 +- .../processors/cache/persistence/DataStorageMetricsImpl.java | 2 +- .../processors/cache/persistence/DataStorageMetricsSnapshot.java | 4 ++-- .../ignite/internal/visor/node/VisorPersistenceMetrics.java | 2 +- .../java/org/apache/ignite/mxbean/DataStorageMetricsMXBean.java | 2 +- 6 files changed, 7 insertions(+), 7 deletions(-) -- http://git-wip-us.apache.org/repos/asf/ignite/blob/6c58b4ac/modules/core/src/main/java/org/apache/ignite/DataStorageMetrics.java -- diff --git a/modules/core/src/main/java/org/apache/ignite/DataStorageMetrics.java b/modules/core/src/main/java/org/apache/ignite/DataStorageMetrics.java index 87095f6..e26bb1f 100644 --- a/modules/core/src/main/java/org/apache/ignite/DataStorageMetrics.java +++ b/modules/core/src/main/java/org/apache/ignite/DataStorageMetrics.java @@ -61,7 +61,7 @@ public interface DataStorageMetrics { * * @return Total checkpoint duration in milliseconds. */ -public long getLastCheckpointingDuration(); +public long getLastCheckpointDuration(); /** * Gets the duration of last checkpoint lock wait in milliseconds. http://git-wip-us.apache.org/repos/asf/ignite/blob/6c58b4ac/modules/core/src/main/java/org/apache/ignite/DataStorageMetricsAdapter.java -- diff --git a/modules/core/src/main/java/org/apache/ignite/DataStorageMetricsAdapter.java b/modules/core/src/main/java/org/apache/ignite/DataStorageMetricsAdapter.java index 6bb4b7e..431e264 100644 --- a/modules/core/src/main/java/org/apache/ignite/DataStorageMetricsAdapter.java +++ b/modules/core/src/main/java/org/apache/ignite/DataStorageMetricsAdapter.java @@ -61,7 +61,7 @@ public class DataStorageMetricsAdapter implements PersistenceMetrics { /** {@inheritDoc} */ @Override public long getLastCheckpointingDuration() { -return delegate.getLastCheckpointingDuration(); +return delegate.getLastCheckpointDuration(); } /** {@inheritDoc} */ http://git-wip-us.apache.org/repos/asf/ignite/blob/6c58b4ac/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/DataStorageMetricsImpl.java -- diff --git a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/DataStorageMetricsImpl.java b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/DataStorageMetricsImpl.java index 16707aa..6d196dc 100644 --- a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/DataStorageMetricsImpl.java +++ b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/DataStorageMetricsImpl.java @@ -127,7 +127,7 @@ public class DataStorageMetricsImpl implements DataStorageMetricsMXBean { } /** {@inheritDoc} */ -@Override public long getLastCheckpointingDuration() { +@Override public long getLastCheckpointDuration() { if (!metricsEnabled) return 0; http://git-wip-us.apache.org/repos/asf/ignite/blob/6c58b4ac/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/DataStorageMetricsSnapshot.java -- diff --git a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/DataStorageMetricsSnapshot.java b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/DataStorageMetricsSnapshot.java index 4841387..5bbb0e1 100644 --- a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/DataStorageMetricsSnapshot.java +++ b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/DataStorageMetricsSnapshot.java @@ -67,7 +67,7 @@ public class DataStorageMetricsSnapshot implements DataStorageMetrics { walWritingRate = metrics.getWalWritingRate(); walArchiveSegments = metri
ignite git commit: IGNITE-6030 Fixed misspelled metric
Repository: ignite Updated Branches: refs/heads/master ec41370c4 -> ec9a945a1 IGNITE-6030 Fixed misspelled metric Project: http://git-wip-us.apache.org/repos/asf/ignite/repo Commit: http://git-wip-us.apache.org/repos/asf/ignite/commit/ec9a945a Tree: http://git-wip-us.apache.org/repos/asf/ignite/tree/ec9a945a Diff: http://git-wip-us.apache.org/repos/asf/ignite/diff/ec9a945a Branch: refs/heads/master Commit: ec9a945a1b9bb14b12359ff183014287faee8f87 Parents: ec41370 Author: Alexey Goncharuk Authored: Fri Oct 20 13:43:44 2017 +0300 Committer: Alexey Goncharuk Committed: Fri Oct 20 13:43:44 2017 +0300 -- .../core/src/main/java/org/apache/ignite/DataStorageMetrics.java | 2 +- .../main/java/org/apache/ignite/DataStorageMetricsAdapter.java | 2 +- .../processors/cache/persistence/DataStorageMetricsImpl.java | 2 +- .../processors/cache/persistence/DataStorageMetricsSnapshot.java | 4 ++-- .../ignite/internal/visor/node/VisorPersistenceMetrics.java | 2 +- .../java/org/apache/ignite/mxbean/DataStorageMetricsMXBean.java | 2 +- 6 files changed, 7 insertions(+), 7 deletions(-) -- http://git-wip-us.apache.org/repos/asf/ignite/blob/ec9a945a/modules/core/src/main/java/org/apache/ignite/DataStorageMetrics.java -- diff --git a/modules/core/src/main/java/org/apache/ignite/DataStorageMetrics.java b/modules/core/src/main/java/org/apache/ignite/DataStorageMetrics.java index 87095f6..e26bb1f 100644 --- a/modules/core/src/main/java/org/apache/ignite/DataStorageMetrics.java +++ b/modules/core/src/main/java/org/apache/ignite/DataStorageMetrics.java @@ -61,7 +61,7 @@ public interface DataStorageMetrics { * * @return Total checkpoint duration in milliseconds. */ -public long getLastCheckpointingDuration(); +public long getLastCheckpointDuration(); /** * Gets the duration of last checkpoint lock wait in milliseconds. http://git-wip-us.apache.org/repos/asf/ignite/blob/ec9a945a/modules/core/src/main/java/org/apache/ignite/DataStorageMetricsAdapter.java -- diff --git a/modules/core/src/main/java/org/apache/ignite/DataStorageMetricsAdapter.java b/modules/core/src/main/java/org/apache/ignite/DataStorageMetricsAdapter.java index 6bb4b7e..431e264 100644 --- a/modules/core/src/main/java/org/apache/ignite/DataStorageMetricsAdapter.java +++ b/modules/core/src/main/java/org/apache/ignite/DataStorageMetricsAdapter.java @@ -61,7 +61,7 @@ public class DataStorageMetricsAdapter implements PersistenceMetrics { /** {@inheritDoc} */ @Override public long getLastCheckpointingDuration() { -return delegate.getLastCheckpointingDuration(); +return delegate.getLastCheckpointDuration(); } /** {@inheritDoc} */ http://git-wip-us.apache.org/repos/asf/ignite/blob/ec9a945a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/DataStorageMetricsImpl.java -- diff --git a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/DataStorageMetricsImpl.java b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/DataStorageMetricsImpl.java index 16707aa..6d196dc 100644 --- a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/DataStorageMetricsImpl.java +++ b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/DataStorageMetricsImpl.java @@ -127,7 +127,7 @@ public class DataStorageMetricsImpl implements DataStorageMetricsMXBean { } /** {@inheritDoc} */ -@Override public long getLastCheckpointingDuration() { +@Override public long getLastCheckpointDuration() { if (!metricsEnabled) return 0; http://git-wip-us.apache.org/repos/asf/ignite/blob/ec9a945a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/DataStorageMetricsSnapshot.java -- diff --git a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/DataStorageMetricsSnapshot.java b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/DataStorageMetricsSnapshot.java index 4841387..5bbb0e1 100644 --- a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/DataStorageMetricsSnapshot.java +++ b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/DataStorageMetricsSnapshot.java @@ -67,7 +67,7 @@ public class DataStorageMetricsSnapshot implements DataStorageMetrics { walWritingRate = metrics.getWalWritingRate(); walArchiveSegments = metrics.getWa
[11/11] ignite git commit: IGNITE-6030 Allow enabling persistence per data region
IGNITE-6030 Allow enabling persistence per data region Project: http://git-wip-us.apache.org/repos/asf/ignite/repo Commit: http://git-wip-us.apache.org/repos/asf/ignite/commit/74567763 Tree: http://git-wip-us.apache.org/repos/asf/ignite/tree/74567763 Diff: http://git-wip-us.apache.org/repos/asf/ignite/diff/74567763 Branch: refs/heads/ignite-2.3 Commit: 745677631d260cb51cb601ae38af8528aa5d5c66 Parents: 008d870 Author: Ivan Rakov Authored: Fri Oct 20 10:29:57 2017 +0300 Committer: Alexey Goncharuk Committed: Fri Oct 20 10:33:19 2017 +0300 -- examples/config/example-data-regions.xml| 106 +++ examples/config/example-memory-policies.xml | 108 --- .../example-persistent-store.xml| 14 +- .../examples/datagrid/DataRegionsExample.java | 113 +++ .../datagrid/MemoryPoliciesExample.java | 114 --- .../ignite/examples/CacheExamplesSelfTest.java | 6 +- .../benchmarks/jmh/tree/BPlusTreeBenchmark.java | 8 +- .../DummyPersistenceCompatibilityTest.java | 8 +- .../org/apache/ignite/DataRegionMetrics.java| 119 +++ .../apache/ignite/DataRegionMetricsAdapter.java | 106 +++ .../org/apache/ignite/DataStorageMetrics.java | 114 +++ .../ignite/DataStorageMetricsAdapter.java | 101 +++ .../src/main/java/org/apache/ignite/Ignite.java | 48 +- .../apache/ignite/IgniteSystemProperties.java | 4 +- .../java/org/apache/ignite/MemoryMetrics.java | 11 +- .../org/apache/ignite/PersistenceMetrics.java | 4 +- .../configuration/CacheConfiguration.java | 36 +- .../configuration/DataPageEvictionMode.java | 8 +- .../configuration/DataRegionConfiguration.java | 406 + .../configuration/DataStorageConfiguration.java | 882 +++ .../configuration/IgniteConfiguration.java | 46 +- .../configuration/MemoryConfiguration.java | 9 +- .../MemoryPolicyConfiguration.java | 32 +- .../PersistentStoreConfiguration.java | 4 +- .../org/apache/ignite/igfs/IgfsMetrics.java | 4 +- .../apache/ignite/internal/IgniteKernal.java| 77 +- .../ignite/internal/IgniteNodeAttributes.java | 4 + .../org/apache/ignite/internal/IgnitionEx.java | 132 ++- .../ignite/internal/MarshallerContextImpl.java | 3 +- .../discovery/GridDiscoveryManager.java | 31 +- .../pagemem/impl/PageMemoryNoStoreImpl.java | 32 +- .../cache/CacheAffinitySharedManager.java | 4 +- .../processors/cache/CacheGroupContext.java | 29 +- .../processors/cache/CacheGroupData.java| 17 +- .../processors/cache/CacheGroupDescriptor.java | 19 +- .../processors/cache/ClusterCachesInfo.java | 13 +- .../processors/cache/GridCacheAdapter.java | 3 +- .../processors/cache/GridCacheContext.java | 8 +- .../processors/cache/GridCacheMapEntry.java | 10 +- .../processors/cache/GridCacheProcessor.java| 55 +- .../processors/cache/GridCacheUtils.java| 55 ++ .../cache/IgniteCacheOffheapManagerImpl.java| 6 +- .../cache/binary/BinaryMetadataFileStore.java | 7 +- .../GridDistributedTxRemoteAdapter.java | 5 +- .../distributed/dht/GridDhtLocalPartition.java | 2 +- .../dht/atomic/GridDhtAtomicCache.java | 3 +- .../distributed/near/GridNearGetRequest.java| 2 +- .../distributed/near/GridNearLockRequest.java | 2 +- .../cache/distributed/near/GridNearTxLocal.java | 2 +- .../near/GridNearTxPrepareRequest.java | 2 +- .../local/atomic/GridLocalAtomicCache.java | 2 +- .../cache/persistence/CacheDataRowAdapter.java | 2 +- .../cache/persistence/DataRegion.java | 84 ++ .../persistence/DataRegionMetricsImpl.java | 286 ++ .../DataRegionMetricsMXBeanImpl.java| 131 +++ .../persistence/DataRegionMetricsSnapshot.java | 112 +++ .../persistence/DataStorageMetricsImpl.java | 297 +++ .../persistence/DataStorageMetricsSnapshot.java | 144 +++ .../GridCacheDatabaseSharedManager.java | 166 ++-- .../persistence/GridCacheOffheapManager.java| 22 +- .../IgniteCacheDatabaseSharedManager.java | 538 +-- .../cache/persistence/MemoryMetricsImpl.java| 286 -- .../persistence/MemoryMetricsMXBeanImpl.java| 131 --- .../persistence/MemoryMetricsSnapshot.java | 112 --- .../cache/persistence/MemoryPolicy.java | 84 -- .../persistence/PersistenceMetricsImpl.java | 297 --- .../persistence/PersistenceMetricsSnapshot.java | 144 --- .../processors/cache/persistence/RowStore.java | 38 +- .../evict/FairFifoPageEvictionTracker.java | 6 +- .../evict/PageAbstractEvictionTracker.java | 6 +- .../evict/Random2LruPageEvictionTracker.java| 8 +- .../evict/RandomLruPageEvictionTracker.java | 8 +- .../persistence/file/AsyncFileIOFactory.java| 28 +- .../cache/persistence/file/FilePageStore.java | 6 +- .../persistence/fil
[09/11] ignite git commit: IGNITE-6030 Allow enabling persistence per data region
http://git-wip-us.apache.org/repos/asf/ignite/blob/74567763/modules/core/src/main/java/org/apache/ignite/internal/pagemem/impl/PageMemoryNoStoreImpl.java -- diff --git a/modules/core/src/main/java/org/apache/ignite/internal/pagemem/impl/PageMemoryNoStoreImpl.java b/modules/core/src/main/java/org/apache/ignite/internal/pagemem/impl/PageMemoryNoStoreImpl.java index 8f146dc..6ba68c2 100644 --- a/modules/core/src/main/java/org/apache/ignite/internal/pagemem/impl/PageMemoryNoStoreImpl.java +++ b/modules/core/src/main/java/org/apache/ignite/internal/pagemem/impl/PageMemoryNoStoreImpl.java @@ -27,14 +27,14 @@ import java.util.concurrent.locks.ReentrantReadWriteLock; import org.apache.ignite.IgniteException; import org.apache.ignite.IgniteLogger; import org.apache.ignite.IgniteSystemProperties; -import org.apache.ignite.configuration.MemoryPolicyConfiguration; +import org.apache.ignite.configuration.DataRegionConfiguration; import org.apache.ignite.internal.mem.DirectMemoryProvider; import org.apache.ignite.internal.mem.DirectMemoryRegion; import org.apache.ignite.internal.mem.IgniteOutOfMemoryException; import org.apache.ignite.internal.pagemem.PageIdUtils; import org.apache.ignite.internal.pagemem.PageMemory; import org.apache.ignite.internal.processors.cache.GridCacheSharedContext; -import org.apache.ignite.internal.processors.cache.persistence.MemoryMetricsImpl; +import org.apache.ignite.internal.processors.cache.persistence.DataRegionMetricsImpl; import org.apache.ignite.internal.processors.cache.persistence.tree.io.PageIO; import org.apache.ignite.internal.util.GridUnsafe; import org.apache.ignite.internal.util.IgniteUtils; @@ -125,11 +125,11 @@ public class PageMemoryNoStoreImpl implements PageMemory { /** Direct memory allocator. */ private final DirectMemoryProvider directMemoryProvider; -/** Name of MemoryPolicy this PageMemory is associated with. */ -private final MemoryPolicyConfiguration memoryPolicyCfg; +/** Name of DataRegion this PageMemory is associated with. */ +private final DataRegionConfiguration dataRegionCfg; /** Object to collect memory usage metrics. */ -private final MemoryMetricsImpl memMetrics; +private final DataRegionMetricsImpl memMetrics; /** */ private AtomicLong freePageListHead = new AtomicLong(INVALID_REL_PTR); @@ -163,7 +163,7 @@ public class PageMemoryNoStoreImpl implements PageMemory { * @param directMemoryProvider Memory allocator to use. * @param sharedCtx Cache shared context. * @param pageSize Page size. - * @param memPlcCfg Memory Policy configuration. + * @param dataRegionCfg Data region configuration. * @param memMetrics Memory Metrics. * @param trackAcquiredPages If {@code true} tracks number of allocated pages (for tests purpose only). */ @@ -172,8 +172,8 @@ public class PageMemoryNoStoreImpl implements PageMemory { DirectMemoryProvider directMemoryProvider, GridCacheSharedContext sharedCtx, int pageSize, -MemoryPolicyConfiguration memPlcCfg, -MemoryMetricsImpl memMetrics, +DataRegionConfiguration dataRegionCfg, +DataRegionMetricsImpl memMetrics, boolean trackAcquiredPages ) { assert log != null || sharedCtx != null; @@ -183,21 +183,21 @@ public class PageMemoryNoStoreImpl implements PageMemory { this.directMemoryProvider = directMemoryProvider; this.trackAcquiredPages = trackAcquiredPages; this.memMetrics = memMetrics; -memoryPolicyCfg = memPlcCfg; +this.dataRegionCfg = dataRegionCfg; sysPageSize = pageSize + PAGE_OVERHEAD; assert sysPageSize % 8 == 0 : sysPageSize; -totalPages = (int)(memPlcCfg.getMaxSize() / sysPageSize); +totalPages = (int)(dataRegionCfg.getMaxSize() / sysPageSize); rwLock = new OffheapReadWriteLock(lockConcLvl); } /** {@inheritDoc} */ @Override public void start() throws IgniteException { -long startSize = memoryPolicyCfg.getInitialSize(); -long maxSize = memoryPolicyCfg.getMaxSize(); +long startSize = dataRegionCfg.getInitialSize(); +long maxSize = dataRegionCfg.getMaxSize(); long[] chunks = new long[SEG_CNT]; @@ -290,9 +290,9 @@ public class PageMemoryNoStoreImpl implements PageMemory { if (relPtr == INVALID_REL_PTR) throw new IgniteOutOfMemoryException("Not enough memory allocated " + -"(consider increasing memory policy size or enabling evictions) " + -"[policyName=" + memoryPolicyCfg.getName() + -", size=" + U.readableSize(memoryPolicyCfg.getMaxSize(), true) + "]" +"(consider increasing data region size or enabling evictions) " + +"[policyName=" + dataRegionCfg.getName() + +", size=" + U.readableSize(d
[01/11] ignite git commit: IGNITE-6030 Allow enabling persistence per data region
Repository: ignite Updated Branches: refs/heads/ignite-2.3 008d87057 -> 745677631 http://git-wip-us.apache.org/repos/asf/ignite/blob/74567763/modules/web-console/frontend/app/modules/states/configuration/clusters/data-storage.pug -- diff --git a/modules/web-console/frontend/app/modules/states/configuration/clusters/data-storage.pug b/modules/web-console/frontend/app/modules/states/configuration/clusters/data-storage.pug new file mode 100644 index 000..9c2dca1 --- /dev/null +++ b/modules/web-console/frontend/app/modules/states/configuration/clusters/data-storage.pug @@ -0,0 +1,255 @@ +//- +Licensed to the Apache Software Foundation (ASF) under one or more +contributor license agreements. See the NOTICE file distributed with +this work for additional information regarding copyright ownership. +The ASF licenses this file to You under the Apache License, Version 2.0 +(the "License"); you may not use this file except in compliance with +the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +include /app/helpers/jade/mixins + +-var form = 'dataStorageConfiguration' +-var model = 'backupItem.dataStorageConfiguration' +-var dfltRegionModel = model + '.defaultDataRegionConfiguration' +-var dataRegionConfigurations = model + '.dataRegionConfigurations' + +.panel.panel-default(ng-show='$ctrl.available("2.3.0")' ng-form=form novalidate) +.panel-heading(bs-collapse-toggle ng-click=`ui.loadPanel('${form}')`) +ignite-form-panel-chevron +label Data storage configuration +ignite-form-field-tooltip.tipLabel +| Page memory is a manageable off-heap based memory architecture that is split into pages of fixed size#[br] +| #[a(href="https://apacheignite.readme.io/docs/distributed-persistent-store"; target="_blank") More info] +ignite-form-revert +.panel-collapse(role='tabpanel' bs-collapse-target id=`${form}`) +.panel-body(ng-if=`$ctrl.available("2.3.0") && ui.isPanelLoaded('${form}')`) +.col-sm-6 +.settings-row ++number-min-max('Page size:', model + '.pageSize', '"DataStorageConfigurationPageSize"', +'true', '2048', '1024', '16384', 'Every memory region is split on pages of fixed size') +.settings-row ++number('Concurrency level:', model + '.concurrencyLevel', '"DataStorageConfigurationConcurrencyLevel"', +'true', 'availableProcessors', '2', 'The number of concurrent segments in Ignite internal page mapping tables') +.settings-row ++ignite-form-group +ignite-form-field-label +| System region +ignite-form-group-tooltip +| System region properties +.group-content +.details-row ++number('Initial size:', model + '.systemRegionInitialSize', '"DataStorageSystemRegionInitialSize"', +'true', '41943040', '10485760', 'Initial size of a data region reserved for system cache') +.details-row ++number('Maximum size:', model + '.systemRegionMaxSize', '"DataStorageSystemRegionMaxSize"', +'true', '104857600', '10485760', 'Maximum data region size reserved for system cache') +.settings-row ++ignite-form-group +ignite-form-field-label +| Data regions +ignite-form-group-tooltip +| Data region configurations +.group-content +.details-row ++ignite-form-group +ignite-form-field-label +| Default data region +ignite-form-group-tooltip +| Default data region properties +.group-content +.details-row ++text('Name:', dfltRegionModel + '.name', '"DfltRegionName" + $index', 'false', 'default', 'Default data region name') +.details-row ++number('Initial siz
[05/11] ignite git commit: IGNITE-6030 Allow enabling persistence per data region
http://git-wip-us.apache.org/repos/asf/ignite/blob/74567763/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/GridMemoryConfigurationConsistencySelfTest.java -- diff --git a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/GridMemoryConfigurationConsistencySelfTest.java b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/GridMemoryConfigurationConsistencySelfTest.java deleted file mode 100644 index bc71e33..000 --- a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/GridMemoryConfigurationConsistencySelfTest.java +++ /dev/null @@ -1,79 +0,0 @@ -/* -* Licensed to the Apache Software Foundation (ASF) under one or more -* contributor license agreements. See the NOTICE file distributed with -* this work for additional information regarding copyright ownership. -* The ASF licenses this file to You under the Apache License, Version 2.0 -* (the "License"); you may not use this file except in compliance with -* the License. You may obtain a copy of the License at -* -* http://www.apache.org/licenses/LICENSE-2.0 -* -* Unless required by applicable law or agreed to in writing, software -* distributed under the License is distributed on an "AS IS" BASIS, -* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -* See the License for the specific language governing permissions and -* limitations under the License. -*/ - -package org.apache.ignite.internal.processors.cache; - -import java.util.concurrent.Callable; -import org.apache.ignite.IgniteCheckedException; -import org.apache.ignite.configuration.IgniteConfiguration; -import org.apache.ignite.configuration.MemoryConfiguration; -import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi; -import org.apache.ignite.spi.discovery.tcp.ipfinder.TcpDiscoveryIpFinder; -import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder; -import org.apache.ignite.testframework.GridTestUtils; -import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest; - -/** - * Tests a check of memory configuration consistency. - */ -public class GridMemoryConfigurationConsistencySelfTest extends GridCommonAbstractTest { -/** IP finder. */ -private static final TcpDiscoveryIpFinder IP_FINDER = new TcpDiscoveryVmIpFinder(true); - -/** {@inheritDoc} */ -@Override protected IgniteConfiguration getConfiguration(String gridName) throws Exception { -IgniteConfiguration cfg = super.getConfiguration(gridName); - -TcpDiscoverySpi discoSpi = new TcpDiscoverySpi(); -discoSpi.setIpFinder(IP_FINDER); - -cfg.setDiscoverySpi(discoSpi); - -MemoryConfiguration memCfg = new MemoryConfiguration(); - -// Nodes will have different page size. -memCfg.setPageSize(MemoryConfiguration.DFLT_PAGE_SIZE * (1 + getTestIgniteInstanceIndex(gridName))); - -cfg.setMemoryConfiguration(memCfg); - -return cfg; -} - -/** - * @throws Exception If failed. - */ -public void testMemoryConfigurationConsistency() throws Exception { -GridTestUtils.assertThrows(log, new Callable() { -/** {@inheritDoc} */ -@Override public Void call() throws Exception { -startGrids(2); - -return null; -} -}, IgniteCheckedException.class, null); -} - -/** {@inheritDoc} */ -@Override protected void beforeTest() throws Exception { -stopAllGrids(); -} - -/** {@inheritDoc} */ -@Override protected void afterTest() throws Exception { -stopAllGrids(); -} -} http://git-wip-us.apache.org/repos/asf/ignite/blob/74567763/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/IgniteClusterActivateDeactivateTest.java -- diff --git a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/IgniteClusterActivateDeactivateTest.java b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/IgniteClusterActivateDeactivateTest.java index 4c9ad27..1827c65 100644 --- a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/IgniteClusterActivateDeactivateTest.java +++ b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/IgniteClusterActivateDeactivateTest.java @@ -29,9 +29,9 @@ import org.apache.ignite.IgniteCache; import org.apache.ignite.cache.CacheAtomicityMode; import org.apache.ignite.cluster.ClusterNode; import org.apache.ignite.configuration.CacheConfiguration; +import org.apache.ignite.configuration.DataRegionConfiguration; +import org.apache.ignite.configuration.DataStorageConfiguration; import org.apache.ignite.configuration.IgniteConfiguration; -import org.apache.ignite.configuration.MemoryConfiguration; -import org.apache.ignite.configuration.PersistentStoreCon
[07/11] ignite git commit: IGNITE-6030 Allow enabling persistence per data region
http://git-wip-us.apache.org/repos/asf/ignite/blob/74567763/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/MemoryMetricsSnapshot.java -- diff --git a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/MemoryMetricsSnapshot.java b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/MemoryMetricsSnapshot.java deleted file mode 100644 index 4e7f90a..000 --- a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/MemoryMetricsSnapshot.java +++ /dev/null @@ -1,112 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.ignite.internal.processors.cache.persistence; - -import org.apache.ignite.MemoryMetrics; - -/** - * - */ -public class MemoryMetricsSnapshot implements MemoryMetrics { -/** */ -private String name; - -/** */ -private long totalAllocatedPages; - -/** */ -private float allocationRate; - -/** */ -private float evictionRate; - -/** */ -private float largeEntriesPagesPercentage; - -/** */ -private float pagesFillFactor; - -/** */ -private long dirtyPages; - -/** */ -private float pageReplaceRate; - -/** */ -private long physicalMemoryPages; - -/** - * @param metrics Metrics instance to take a copy. - */ -public MemoryMetricsSnapshot(MemoryMetrics metrics) { -name = metrics.getName(); -totalAllocatedPages = metrics.getTotalAllocatedPages(); -allocationRate = metrics.getAllocationRate(); -evictionRate = metrics.getEvictionRate(); -largeEntriesPagesPercentage = metrics.getLargeEntriesPagesPercentage(); -pagesFillFactor = metrics.getPagesFillFactor(); -dirtyPages = metrics.getDirtyPages(); -pageReplaceRate = metrics.getPagesReplaceRate(); -physicalMemoryPages = metrics.getPhysicalMemoryPages(); -} - -/** {@inheritDoc} */ -@Override public String getName() { -return name; -} - -/** {@inheritDoc} */ -@Override public long getTotalAllocatedPages() { -return totalAllocatedPages; -} - -/** {@inheritDoc} */ -@Override public float getAllocationRate() { -return allocationRate; -} - -/** {@inheritDoc} */ -@Override public float getEvictionRate() { -return evictionRate; -} - -/** {@inheritDoc} */ -@Override public float getLargeEntriesPagesPercentage() { -return largeEntriesPagesPercentage; -} - -/** {@inheritDoc} */ -@Override public float getPagesFillFactor() { -return pagesFillFactor; -} - -/** {@inheritDoc} */ -@Override public long getDirtyPages() { -return dirtyPages; -} - -/** {@inheritDoc} */ -@Override public float getPagesReplaceRate() { -return pageReplaceRate; -} - -/** {@inheritDoc} */ -@Override public long getPhysicalMemoryPages() { -return physicalMemoryPages; -} -} http://git-wip-us.apache.org/repos/asf/ignite/blob/74567763/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/MemoryPolicy.java -- diff --git a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/MemoryPolicy.java b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/MemoryPolicy.java deleted file mode 100644 index 4059c12..000 --- a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/MemoryPolicy.java +++ /dev/null @@ -1,84 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless
[03/11] ignite git commit: IGNITE-6030 Allow enabling persistence per data region
http://git-wip-us.apache.org/repos/asf/ignite/blob/74567763/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/wal/reader/IgniteWalReaderTest.java -- diff --git a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/wal/reader/IgniteWalReaderTest.java b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/wal/reader/IgniteWalReaderTest.java index 6db2784..4a4010a 100644 --- a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/wal/reader/IgniteWalReaderTest.java +++ b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/wal/reader/IgniteWalReaderTest.java @@ -45,10 +45,9 @@ import org.apache.ignite.cache.CacheAtomicityMode; import org.apache.ignite.cache.CacheRebalanceMode; import org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction; import org.apache.ignite.configuration.CacheConfiguration; +import org.apache.ignite.configuration.DataStorageConfiguration; import org.apache.ignite.configuration.IgniteConfiguration; -import org.apache.ignite.configuration.MemoryConfiguration; -import org.apache.ignite.configuration.MemoryPolicyConfiguration; -import org.apache.ignite.configuration.PersistentStoreConfiguration; +import org.apache.ignite.configuration.DataRegionConfiguration; import org.apache.ignite.configuration.WALMode; import org.apache.ignite.events.Event; import org.apache.ignite.events.EventType; @@ -130,31 +129,20 @@ public class IgniteWalReaderTest extends GridCommonAbstractTest { cfg.setIncludeEventTypes(EventType.EVT_WAL_SEGMENT_ARCHIVED); -final MemoryConfiguration dbCfg = new MemoryConfiguration(); - -dbCfg.setPageSize(PAGE_SIZE); - -final MemoryPolicyConfiguration memPlcCfg = new MemoryPolicyConfiguration(); - -memPlcCfg.setName("dfltMemPlc"); -memPlcCfg.setInitialSize(1024 * 1024 * 1024); -memPlcCfg.setMaxSize(1024 * 1024 * 1024); - -dbCfg.setMemoryPolicies(memPlcCfg); -dbCfg.setDefaultMemoryPolicyName("dfltMemPlc"); - -cfg.setMemoryConfiguration(dbCfg); - -final PersistentStoreConfiguration pCfg = new PersistentStoreConfiguration(); -pCfg.setWalHistorySize(1); -pCfg.setWalSegmentSize(1024 * 1024); -pCfg.setWalSegments(WAL_SEGMENTS); -pCfg.setWalMode(customWalMode != null ? customWalMode : WALMode.BACKGROUND); +DataStorageConfiguration memCfg = new DataStorageConfiguration() +.setDefaultDataRegionConfiguration( +new DataRegionConfiguration().setMaxSize(1024 * 1024 * 1024).setPersistenceEnabled(true)) +.setPageSize(PAGE_SIZE) +.setWalHistorySize(1) +.setWalSegmentSize(1024 * 1024) +.setWalSegments(WAL_SEGMENTS) +.setWalMode(customWalMode != null ? customWalMode : WALMode.BACKGROUND); if (archiveIncompleteSegmentAfterInactivityMs > 0) - pCfg.setWalAutoArchiveAfterInactivity(archiveIncompleteSegmentAfterInactivityMs); + memCfg.setWalAutoArchiveAfterInactivity(archiveIncompleteSegmentAfterInactivityMs); + +cfg.setDataStorageConfiguration(memCfg); -cfg.setPersistentStoreConfiguration(pCfg); return cfg; } @@ -492,8 +480,8 @@ public class IgniteWalReaderTest extends GridCommonAbstractTest { * @param factory WAL iterator factory. * @param workDir Ignite work directory. * @param subfolderName DB subfolder name based on consistent ID. - * @param expCntEntries minimum expected entries count to find. - * @param expTxCnt minimum expected transaction count to find. + * @param minCntEntries minimum expected entries count to find. + * @param minTxCnt minimum expected transaction count to find. * @param objConsumer object handler, called for each object found in logical data records. * @param dataRecordHnd data handler record * @throws IgniteCheckedException if failed. @@ -502,8 +490,8 @@ public class IgniteWalReaderTest extends GridCommonAbstractTest { final IgniteWalIteratorFactory factory, final String workDir, final String subfolderName, -final int expCntEntries, -final int expTxCnt, +final int minCntEntries, +final int minTxCnt, @Nullable final BiConsumer objConsumer, @Nullable final Consumer dataRecordHnd) throws IgniteCheckedException { @@ -538,8 +526,8 @@ public class IgniteWalReaderTest extends GridCommonAbstractTest { final int entriesWork = valuesSum(cntWork.values()); log.info("Archive directory: Tx found " + txCntObservedWork + " entries " + entriesWork); -assert entriesArch + entriesWork >= expCntEntries; -assert txCntObservedWork + txCntObservedArch >= expTxCnt; +asse
[06/11] ignite git commit: IGNITE-6030 Allow enabling persistence per data region
http://git-wip-us.apache.org/repos/asf/ignite/blob/74567763/modules/core/src/main/java/org/apache/ignite/internal/visor/node/VisorDataRegionConfiguration.java -- diff --git a/modules/core/src/main/java/org/apache/ignite/internal/visor/node/VisorDataRegionConfiguration.java b/modules/core/src/main/java/org/apache/ignite/internal/visor/node/VisorDataRegionConfiguration.java new file mode 100644 index 000..394e294 --- /dev/null +++ b/modules/core/src/main/java/org/apache/ignite/internal/visor/node/VisorDataRegionConfiguration.java @@ -0,0 +1,225 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.ignite.internal.visor.node; + +import java.io.IOException; +import java.io.ObjectInput; +import java.io.ObjectOutput; +import java.util.ArrayList; +import java.util.List; +import org.apache.ignite.configuration.DataPageEvictionMode; +import org.apache.ignite.configuration.DataRegionConfiguration; +import org.apache.ignite.internal.util.typedef.internal.S; +import org.apache.ignite.internal.util.typedef.internal.U; +import org.apache.ignite.internal.visor.VisorDataTransferObject; + +/** + * Data transfer object for data region configuration. + */ +public class VisorDataRegionConfiguration extends VisorDataTransferObject { +/** */ +private static final long serialVersionUID = 0L; + +/** Unique name of DataRegion. */ +private String name; + +/** Initial memory region size defined by this memory policy. */ +private long initSize; + +/** Maximum memory region size defined by this memory policy. */ +private long maxSize; + +/** Path for memory mapped file. */ +private String swapPath; + +/** An algorithm for memory pages eviction. */ +private DataPageEvictionMode pageEvictionMode; + +/** A threshold for memory pages eviction initiation. */ +private double evictionThreshold; + +/** Minimum number of empty pages in reuse lists. */ +private int emptyPagesPoolSize; + +/** Enable memory metrics collection for this data region. */ +private boolean metricsEnabled; + +/** Number of sub-intervals. */ +private int metricsSubIntervalCount; + +/** Time interval over which allocation rate is calculated. */ +private long metricsRateTimeInterval; + +/** Enable Ignite Native Persistence. */ +private boolean persistenceEnabled; + +/** + * Default constructor. + */ +public VisorDataRegionConfiguration() { +// No-op. +} + +/** + * Constructor. + * + * @param plc Data region configuration. + */ +public VisorDataRegionConfiguration(DataRegionConfiguration plc) { +assert plc != null; + +name = plc.getName(); +initSize = plc.getInitialSize(); +maxSize = plc.getMaxSize(); +swapPath = plc.getSwapPath(); +pageEvictionMode = plc.getPageEvictionMode(); +evictionThreshold = plc.getEvictionThreshold(); +emptyPagesPoolSize = plc.getEmptyPagesPoolSize(); +metricsEnabled = plc.isMetricsEnabled(); +metricsSubIntervalCount = plc.getMetricsSubIntervalCount(); +metricsRateTimeInterval = plc.getMetricsRateTimeInterval(); +persistenceEnabled = plc.isPersistenceEnabled(); +} + +/** + * @param regCfgs Array of data region configurations. + * @return Collection of DTO objects. + */ +public static List from(DataRegionConfiguration[] regCfgs) { +List res = new ArrayList<>(); + +if (regCfgs != null) { +for (DataRegionConfiguration plc: regCfgs) +res.add(new VisorDataRegionConfiguration(plc)); +} + +return res; +} + +/** + * @return Unique name of DataRegion. + */ +public String getName() { +return name; +} + +/** + * @return Maximum memory region size defined by this memory policy. + */ +public long getMaxSize() { +return maxSize; +} + +/** + * @return Initial memory region size defined by this memory policy. + */ +public long getInitialSize() { +return initSize; +} + +/** + * @return Path for memor
[02/11] ignite git commit: IGNITE-6030 Allow enabling persistence per data region
http://git-wip-us.apache.org/repos/asf/ignite/blob/74567763/modules/core/src/test/java/org/apache/ignite/internal/processors/database/MetadataStorageSelfTest.java -- diff --git a/modules/core/src/test/java/org/apache/ignite/internal/processors/database/MetadataStorageSelfTest.java b/modules/core/src/test/java/org/apache/ignite/internal/processors/database/MetadataStorageSelfTest.java index dcd4ce1..880e37e 100644 --- a/modules/core/src/test/java/org/apache/ignite/internal/processors/database/MetadataStorageSelfTest.java +++ b/modules/core/src/test/java/org/apache/ignite/internal/processors/database/MetadataStorageSelfTest.java @@ -23,12 +23,12 @@ import java.util.Map; import java.util.Random; import java.util.concurrent.ThreadLocalRandom; import java.util.concurrent.atomic.AtomicLong; -import org.apache.ignite.configuration.MemoryPolicyConfiguration; +import org.apache.ignite.configuration.DataRegionConfiguration; import org.apache.ignite.internal.mem.DirectMemoryProvider; import org.apache.ignite.internal.pagemem.FullPageId; import org.apache.ignite.internal.pagemem.PageIdAllocator; import org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl; -import org.apache.ignite.internal.processors.cache.persistence.MemoryMetricsImpl; +import org.apache.ignite.internal.processors.cache.persistence.DataRegionMetricsImpl; import org.apache.ignite.internal.processors.cache.persistence.MetadataStorage; import org.apache.ignite.internal.mem.file.MappedFileMemoryProvider; import org.apache.ignite.internal.pagemem.PageMemory; @@ -156,7 +156,7 @@ public class MetadataStorageSelfTest extends GridCommonAbstractTest { protected PageMemory memory(boolean clean) throws Exception { DirectMemoryProvider provider = new MappedFileMemoryProvider(log(), allocationPath); -MemoryPolicyConfiguration plcCfg = new MemoryPolicyConfiguration() +DataRegionConfiguration plcCfg = new DataRegionConfiguration() .setMaxSize(30 * 1024 * 1024).setInitialSize(30 * 1024 * 1024); return new PageMemoryNoStoreImpl( @@ -165,7 +165,7 @@ public class MetadataStorageSelfTest extends GridCommonAbstractTest { null, PAGE_SIZE, plcCfg, -new MemoryMetricsImpl(plcCfg), +new DataRegionMetricsImpl(plcCfg), true); } } http://git-wip-us.apache.org/repos/asf/ignite/blob/74567763/modules/core/src/test/java/org/apache/ignite/internal/processors/database/SwapPathConstructionSelfTest.java -- diff --git a/modules/core/src/test/java/org/apache/ignite/internal/processors/database/SwapPathConstructionSelfTest.java b/modules/core/src/test/java/org/apache/ignite/internal/processors/database/SwapPathConstructionSelfTest.java index 53e5daf..f22128c 100644 --- a/modules/core/src/test/java/org/apache/ignite/internal/processors/database/SwapPathConstructionSelfTest.java +++ b/modules/core/src/test/java/org/apache/ignite/internal/processors/database/SwapPathConstructionSelfTest.java @@ -22,23 +22,23 @@ import java.nio.file.Path; import java.nio.file.Paths; import java.util.Map; import org.apache.ignite.configuration.IgniteConfiguration; -import org.apache.ignite.configuration.MemoryConfiguration; -import org.apache.ignite.configuration.MemoryPolicyConfiguration; +import org.apache.ignite.configuration.DataStorageConfiguration; +import org.apache.ignite.configuration.DataRegionConfiguration; import org.apache.ignite.internal.GridKernalContext; import org.apache.ignite.internal.IgniteEx; import org.apache.ignite.internal.pagemem.PageMemory; import org.apache.ignite.internal.processors.cache.persistence.IgniteCacheDatabaseSharedManager; -import org.apache.ignite.internal.processors.cache.persistence.MemoryPolicy; +import org.apache.ignite.internal.processors.cache.persistence.DataRegion; import org.apache.ignite.internal.util.typedef.internal.U; import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest; /** - * Test verifies correct construction of swap file path {@link MemoryPolicyConfiguration#setSwapFilePath(String)} + * Test verifies correct construction of swap file path {@link DataRegionConfiguration#setSwapPath(String)} * when absolute or relative paths are provided via configuration. */ public class SwapPathConstructionSelfTest extends GridCommonAbstractTest { /** */ -private MemoryConfiguration memCfg; +private DataStorageConfiguration memCfg; /** */ private static final String RELATIVE_SWAP_PATH = "relSwapPath"; @@ -50,7 +50,7 @@ public class SwapPathConstructionSelfTest extends GridCommonAbstractTest { @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception { IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName); -cfg.setMemory
[04/11] ignite git commit: IGNITE-6030 Allow enabling persistence per data region
http://git-wip-us.apache.org/repos/asf/ignite/blob/74567763/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/MemoryPolicyInitializationTest.java -- diff --git a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/MemoryPolicyInitializationTest.java b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/MemoryPolicyInitializationTest.java index 842f618..fb1574d 100644 --- a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/MemoryPolicyInitializationTest.java +++ b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/MemoryPolicyInitializationTest.java @@ -73,7 +73,7 @@ public class MemoryPolicyInitializationTest extends GridCommonAbstractTest { IgniteEx ignite = startGrid(0); -Collection allMemPlcs = ignite.context().cache().context().database().memoryPolicies(); +Collection allMemPlcs = ignite.context().cache().context().database().dataRegions(); assertTrue(allMemPlcs.size() == 2); @@ -89,7 +89,7 @@ public class MemoryPolicyInitializationTest extends GridCommonAbstractTest { IgniteEx ignite = startGrid(0); -Collection allMemPlcs = ignite.context().cache().context().database().memoryPolicies(); +Collection allMemPlcs = ignite.context().cache().context().database().dataRegions(); assertTrue(allMemPlcs.size() == 3); @@ -110,13 +110,13 @@ public class MemoryPolicyInitializationTest extends GridCommonAbstractTest { IgniteCacheDatabaseSharedManager dbMgr = ignite.context().cache().context().database(); -Collection allMemPlcs = dbMgr.memoryPolicies(); +Collection allMemPlcs = dbMgr.dataRegions(); assertTrue(allMemPlcs.size() == 2); verifyDefaultAndSystemMemoryPolicies(allMemPlcs); -MemoryPolicy dfltMemPlc = U.field(dbMgr, "dfltMemPlc"); +DataRegion dfltMemPlc = U.field(dbMgr, "dfltDataRegion"); assertTrue(dfltMemPlc.config().getMaxSize() == USER_DEFAULT_MEM_PLC_SIZE); } @@ -134,13 +134,13 @@ public class MemoryPolicyInitializationTest extends GridCommonAbstractTest { IgniteCacheDatabaseSharedManager dbMgr = ignite.context().cache().context().database(); -Collection allMemPlcs = dbMgr.memoryPolicies(); +Collection allMemPlcs = dbMgr.dataRegions(); assertTrue(allMemPlcs.size() == 3); verifyDefaultAndSystemMemoryPolicies(allMemPlcs); -MemoryPolicy dfltMemPlc = U.field(dbMgr, "dfltMemPlc"); +DataRegion dfltMemPlc = U.field(dbMgr, "dfltDataRegion"); assertTrue(dfltMemPlc.config().getMaxSize() == USER_CUSTOM_MEM_PLC_SIZE); } @@ -220,7 +220,7 @@ public class MemoryPolicyInitializationTest extends GridCommonAbstractTest { private void verifyCacheMemoryPolicy(IgniteCache cache, String plcName) { GridCacheContext ctx = ((IgniteCacheProxy) cache).context(); -assertEquals(plcName, ctx.memoryPolicy().config().getName()); +assertEquals(plcName, ctx.dataRegion().config().getName()); } /** @@ -278,12 +278,12 @@ public class MemoryPolicyInitializationTest extends GridCommonAbstractTest { /** * @param allMemPlcs Collection of all memory policies. */ -private void verifyDefaultAndSystemMemoryPolicies(Collection allMemPlcs) { +private void verifyDefaultAndSystemMemoryPolicies(Collection allMemPlcs) { assertTrue("Default memory policy is not presented", isMemoryPolicyPresented(allMemPlcs, DFLT_MEM_PLC_DEFAULT_NAME)); assertTrue("System memory policy is not presented", -isMemoryPolicyPresented(allMemPlcs, IgniteCacheDatabaseSharedManager.SYSTEM_MEMORY_POLICY_NAME)); +isMemoryPolicyPresented(allMemPlcs, IgniteCacheDatabaseSharedManager.SYSTEM_DATA_REGION_NAME)); } /** @@ -303,8 +303,8 @@ public class MemoryPolicyInitializationTest extends GridCommonAbstractTest { * @param memPlcs Collection of memory policies. * @param nameToVerify Excepted name of memory policy. */ -private boolean isMemoryPolicyPresented(Collection memPlcs, String nameToVerify) { -for (MemoryPolicy memPlc : memPlcs) { +private boolean isMemoryPolicyPresented(Collection memPlcs, String nameToVerify) { +for (DataRegion memPlc : memPlcs) { if (nameToVerify.equals(memPlc.config().getName())) return true; } http://git-wip-us.apache.org/repos/asf/ignite/blob/74567763/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsCacheRestoreTest.java -- diff --git a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/d
[08/11] ignite git commit: IGNITE-6030 Allow enabling persistence per data region
http://git-wip-us.apache.org/repos/asf/ignite/blob/74567763/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheDatabaseSharedManager.java -- diff --git a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheDatabaseSharedManager.java b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheDatabaseSharedManager.java index a56b8f4..596b7b2 100755 --- a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheDatabaseSharedManager.java +++ b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheDatabaseSharedManager.java @@ -57,18 +57,17 @@ import java.util.concurrent.locks.ReentrantReadWriteLock; import java.util.regex.Matcher; import java.util.regex.Pattern; import javax.management.ObjectName; +import org.apache.ignite.DataStorageMetrics; import org.apache.ignite.IgniteCheckedException; import org.apache.ignite.IgniteException; import org.apache.ignite.IgniteLogger; import org.apache.ignite.IgniteSystemProperties; -import org.apache.ignite.PersistenceMetrics; import org.apache.ignite.cluster.ClusterNode; import org.apache.ignite.configuration.CheckpointWriteOrder; import org.apache.ignite.configuration.DataPageEvictionMode; +import org.apache.ignite.configuration.DataRegionConfiguration; +import org.apache.ignite.configuration.DataStorageConfiguration; import org.apache.ignite.configuration.IgniteConfiguration; -import org.apache.ignite.configuration.MemoryConfiguration; -import org.apache.ignite.configuration.MemoryPolicyConfiguration; -import org.apache.ignite.configuration.PersistentStoreConfiguration; import org.apache.ignite.events.DiscoveryEvent; import org.apache.ignite.events.EventType; import org.apache.ignite.internal.GridKernalContext; @@ -139,7 +138,7 @@ import org.apache.ignite.internal.util.typedef.internal.U; import org.apache.ignite.internal.util.worker.GridWorker; import org.apache.ignite.lang.IgniteBiTuple; import org.apache.ignite.lang.IgniteOutClosure; -import org.apache.ignite.mxbean.PersistenceMetricsMXBean; +import org.apache.ignite.mxbean.DataStorageMetricsMXBean; import org.apache.ignite.thread.IgniteThread; import org.apache.ignite.thread.IgniteThreadPoolExecutor; import org.jetbrains.annotations.NotNull; @@ -229,7 +228,7 @@ public class GridCacheDatabaseSharedManager extends IgniteCacheDatabaseSharedMan }; /** */ -private static final String MBEAN_NAME = "PersistenceMetrics"; +private static final String MBEAN_NAME = "DataStorageMetrics"; /** */ private static final String MBEAN_GROUP = "Persistent Store"; @@ -262,7 +261,7 @@ public class GridCacheDatabaseSharedManager extends IgniteCacheDatabaseSharedMan private volatile boolean printCheckpointStats = true; /** Database configuration. */ -private final PersistentStoreConfiguration persistenceCfg; +private final DataStorageConfiguration persistenceCfg; /** */ private final Collection lsnrs = new CopyOnWriteArrayList<>(); @@ -301,7 +300,7 @@ public class GridCacheDatabaseSharedManager extends IgniteCacheDatabaseSharedMan private IgniteCacheSnapshotManager snapshotMgr; /** */ -private PersistenceMetricsImpl persStoreMetrics; +private DataStorageMetricsImpl persStoreMetrics; /** */ private ObjectName persistenceMetricsMbeanName; @@ -318,18 +317,18 @@ public class GridCacheDatabaseSharedManager extends IgniteCacheDatabaseSharedMan public GridCacheDatabaseSharedManager(GridKernalContext ctx) { IgniteConfiguration cfg = ctx.config(); -persistenceCfg = cfg.getPersistentStoreConfiguration(); +persistenceCfg = cfg.getDataStorageConfiguration(); -assert persistenceCfg != null : "PageStore should not be created if persistence is disabled."; +assert persistenceCfg != null; -checkpointFreq = persistenceCfg.getCheckpointingFrequency(); +checkpointFreq = persistenceCfg.getCheckpointFrequency(); lockWaitTime = persistenceCfg.getLockWaitTime(); -persStoreMetrics = new PersistenceMetricsImpl( +persStoreMetrics = new DataStorageMetricsImpl( persistenceCfg.isMetricsEnabled(), -persistenceCfg.getRateTimeInterval(), -persistenceCfg.getSubIntervals() +persistenceCfg.getMetricsRateTimeInterval(), +persistenceCfg.getMetricsSubIntervalCount() ); } @@ -400,12 +399,12 @@ public class GridCacheDatabaseSharedManager extends IgniteCacheDatabaseSharedMan * */ private void initDataBase() { -if (persistenceCfg.getCheckpointingThreads() > 1) +if (persistenceCfg.getCheckpointThreads() > 1) asyncRunner = new IgniteThreadPoolExecutor( "checkpoint-runner",
[10/11] ignite git commit: IGNITE-6030 Allow enabling persistence per data region
http://git-wip-us.apache.org/repos/asf/ignite/blob/74567763/modules/core/src/main/java/org/apache/ignite/configuration/DataRegionConfiguration.java -- diff --git a/modules/core/src/main/java/org/apache/ignite/configuration/DataRegionConfiguration.java b/modules/core/src/main/java/org/apache/ignite/configuration/DataRegionConfiguration.java new file mode 100644 index 000..50edf5c --- /dev/null +++ b/modules/core/src/main/java/org/apache/ignite/configuration/DataRegionConfiguration.java @@ -0,0 +1,406 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.ignite.configuration; + +import java.io.Serializable; +import org.apache.ignite.DataRegionMetrics; +import org.apache.ignite.internal.mem.IgniteOutOfMemoryException; +import org.apache.ignite.mxbean.DataRegionMetricsMXBean; + +import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_DATA_REG_DEFAULT_NAME; + +/** + * This class allows defining custom data regions' configurations with various parameters for Apache Ignite + * page memory (see {@link DataStorageConfiguration}. For each configured data region Apache Ignite instantiates + * respective memory regions with different parameters like maximum size, eviction policy, swapping options, + * persistent mode flag, etc. + * An Apache Ignite cache can be mapped to a particular region using + * {@link CacheConfiguration#setDataRegionName(String)} method. + * Sample configuration below shows how to configure several data regions: + * + * {@code + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * } + * + */ +public final class DataRegionConfiguration implements Serializable { +/** */ +private static final long serialVersionUID = 0L; + +/** Default metrics enabled flag. */ +public static final boolean DFLT_METRICS_ENABLED = false; + +/** Default amount of sub intervals to calculate {@link DataRegionMetrics#getAllocationRate()} metric. */ +public static final int DFLT_SUB_INTERVALS = 5; + +/** Default length of interval over which {@link DataRegionMetrics#getAllocationRate()} metric is calculated. */ +public static final int DFLT_RATE_TIME_INTERVAL_MILLIS = 60_000; + +/** Data region name. */ +private String name = DFLT_DATA_REG_DEFAULT_NAME; + +/** Data region maximum size in memory. */ +private long maxSize = DataStorageConfiguration.DFLT_DATA_REGION_MAX_SIZE; + +/** Data region start size. */ +private long initSize = Math.min( +DataStorageConfiguration.DFLT_DATA_REGION_MAX_SIZE, DataStorageConfiguration.DFLT_DATA_REGION_INITIAL_SIZE); + +/** An optional path to a memory mapped files directory for this data region. */ +private String swapPath; + +/** An algorithm for memory pages eviction. */ +private DataPageEvictionMode pageEvictionMode = DataPageEvictionMode.DISABLED; + +/** + * A threshold for memory pages eviction initiation. For instance, if the threshold is 0.9 it means that the page + * memory will start the eviction only after 90% data region is occupied. + */ +private double evictionThreshold = 0.9; + +/** Minimum number of empty pages in reuse lists. */ +private int emptyPagesPoolSize = 100; + +/** + * Flag to enable the memory metrics collection for this data region. + */ +private boolean metricsEnabled = DFLT_METRICS_ENABLED; + +/** Number of sub-intervals the whole {@link #setMetricsRateTimeInterval(long)} will be split into to calculate + * {@link DataRegionMetrics#getAllocationRate()} and {@link DataRegionMetrics#getEvictionRate()} rates (5 by
[03/11] ignite git commit: IGNITE-6030 Allow enabling persistence per data region
http://git-wip-us.apache.org/repos/asf/ignite/blob/ec41370c/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/wal/reader/IgniteWalReaderTest.java -- diff --git a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/wal/reader/IgniteWalReaderTest.java b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/wal/reader/IgniteWalReaderTest.java index 79387e3..88eacef 100644 --- a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/wal/reader/IgniteWalReaderTest.java +++ b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/wal/reader/IgniteWalReaderTest.java @@ -45,10 +45,9 @@ import org.apache.ignite.cache.CacheAtomicityMode; import org.apache.ignite.cache.CacheRebalanceMode; import org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction; import org.apache.ignite.configuration.CacheConfiguration; +import org.apache.ignite.configuration.DataStorageConfiguration; import org.apache.ignite.configuration.IgniteConfiguration; -import org.apache.ignite.configuration.MemoryConfiguration; -import org.apache.ignite.configuration.MemoryPolicyConfiguration; -import org.apache.ignite.configuration.PersistentStoreConfiguration; +import org.apache.ignite.configuration.DataRegionConfiguration; import org.apache.ignite.configuration.WALMode; import org.apache.ignite.events.Event; import org.apache.ignite.events.EventType; @@ -131,31 +130,20 @@ public class IgniteWalReaderTest extends GridCommonAbstractTest { cfg.setIncludeEventTypes(EventType.EVT_WAL_SEGMENT_ARCHIVED); -final MemoryConfiguration dbCfg = new MemoryConfiguration(); - -dbCfg.setPageSize(PAGE_SIZE); - -final MemoryPolicyConfiguration memPlcCfg = new MemoryPolicyConfiguration(); - -memPlcCfg.setName("dfltMemPlc"); -memPlcCfg.setInitialSize(1024 * 1024 * 1024); -memPlcCfg.setMaxSize(1024 * 1024 * 1024); - -dbCfg.setMemoryPolicies(memPlcCfg); -dbCfg.setDefaultMemoryPolicyName("dfltMemPlc"); - -cfg.setMemoryConfiguration(dbCfg); - -final PersistentStoreConfiguration pCfg = new PersistentStoreConfiguration(); -pCfg.setWalHistorySize(1); -pCfg.setWalSegmentSize(1024 * 1024); -pCfg.setWalSegments(WAL_SEGMENTS); -pCfg.setWalMode(customWalMode != null ? customWalMode : WALMode.BACKGROUND); +DataStorageConfiguration memCfg = new DataStorageConfiguration() +.setDefaultDataRegionConfiguration( +new DataRegionConfiguration().setMaxSize(1024 * 1024 * 1024).setPersistenceEnabled(true)) +.setPageSize(PAGE_SIZE) +.setWalHistorySize(1) +.setWalSegmentSize(1024 * 1024) +.setWalSegments(WAL_SEGMENTS) +.setWalMode(customWalMode != null ? customWalMode : WALMode.BACKGROUND); if (archiveIncompleteSegmentAfterInactivityMs > 0) - pCfg.setWalAutoArchiveAfterInactivity(archiveIncompleteSegmentAfterInactivityMs); + memCfg.setWalAutoArchiveAfterInactivity(archiveIncompleteSegmentAfterInactivityMs); + +cfg.setDataStorageConfiguration(memCfg); -cfg.setPersistentStoreConfiguration(pCfg); return cfg; } @@ -510,8 +498,8 @@ public class IgniteWalReaderTest extends GridCommonAbstractTest { * @param factory WAL iterator factory. * @param workDir Ignite work directory. * @param subfolderName DB subfolder name based on consistent ID. - * @param expCntEntries minimum expected entries count to find. - * @param expTxCnt minimum expected transaction count to find. + * @param minCntEntries minimum expected entries count to find. + * @param minTxCnt minimum expected transaction count to find. * @param objConsumer object handler, called for each object found in logical data records. * @param dataRecordHnd data handler record * @throws IgniteCheckedException if failed. @@ -520,8 +508,8 @@ public class IgniteWalReaderTest extends GridCommonAbstractTest { final IgniteWalIteratorFactory factory, final String workDir, final String subfolderName, -final int expCntEntries, -final int expTxCnt, +final int minCntEntries, +final int minTxCnt, @Nullable final BiConsumer objConsumer, @Nullable final Consumer dataRecordHnd) throws IgniteCheckedException { @@ -556,8 +544,8 @@ public class IgniteWalReaderTest extends GridCommonAbstractTest { final int entriesWork = valuesSum(cntWork.values()); log.info("Archive directory: Tx found " + txCntObservedWork + " entries " + entriesWork); -assert entriesArch + entriesWork >= expCntEntries; -assert txCntObservedWork + txCntObservedArch >= expTxCnt; +asse
[09/11] ignite git commit: IGNITE-6030 Allow enabling persistence per data region
http://git-wip-us.apache.org/repos/asf/ignite/blob/ec41370c/modules/core/src/main/java/org/apache/ignite/internal/pagemem/impl/PageMemoryNoStoreImpl.java -- diff --git a/modules/core/src/main/java/org/apache/ignite/internal/pagemem/impl/PageMemoryNoStoreImpl.java b/modules/core/src/main/java/org/apache/ignite/internal/pagemem/impl/PageMemoryNoStoreImpl.java index 8f146dc..6ba68c2 100644 --- a/modules/core/src/main/java/org/apache/ignite/internal/pagemem/impl/PageMemoryNoStoreImpl.java +++ b/modules/core/src/main/java/org/apache/ignite/internal/pagemem/impl/PageMemoryNoStoreImpl.java @@ -27,14 +27,14 @@ import java.util.concurrent.locks.ReentrantReadWriteLock; import org.apache.ignite.IgniteException; import org.apache.ignite.IgniteLogger; import org.apache.ignite.IgniteSystemProperties; -import org.apache.ignite.configuration.MemoryPolicyConfiguration; +import org.apache.ignite.configuration.DataRegionConfiguration; import org.apache.ignite.internal.mem.DirectMemoryProvider; import org.apache.ignite.internal.mem.DirectMemoryRegion; import org.apache.ignite.internal.mem.IgniteOutOfMemoryException; import org.apache.ignite.internal.pagemem.PageIdUtils; import org.apache.ignite.internal.pagemem.PageMemory; import org.apache.ignite.internal.processors.cache.GridCacheSharedContext; -import org.apache.ignite.internal.processors.cache.persistence.MemoryMetricsImpl; +import org.apache.ignite.internal.processors.cache.persistence.DataRegionMetricsImpl; import org.apache.ignite.internal.processors.cache.persistence.tree.io.PageIO; import org.apache.ignite.internal.util.GridUnsafe; import org.apache.ignite.internal.util.IgniteUtils; @@ -125,11 +125,11 @@ public class PageMemoryNoStoreImpl implements PageMemory { /** Direct memory allocator. */ private final DirectMemoryProvider directMemoryProvider; -/** Name of MemoryPolicy this PageMemory is associated with. */ -private final MemoryPolicyConfiguration memoryPolicyCfg; +/** Name of DataRegion this PageMemory is associated with. */ +private final DataRegionConfiguration dataRegionCfg; /** Object to collect memory usage metrics. */ -private final MemoryMetricsImpl memMetrics; +private final DataRegionMetricsImpl memMetrics; /** */ private AtomicLong freePageListHead = new AtomicLong(INVALID_REL_PTR); @@ -163,7 +163,7 @@ public class PageMemoryNoStoreImpl implements PageMemory { * @param directMemoryProvider Memory allocator to use. * @param sharedCtx Cache shared context. * @param pageSize Page size. - * @param memPlcCfg Memory Policy configuration. + * @param dataRegionCfg Data region configuration. * @param memMetrics Memory Metrics. * @param trackAcquiredPages If {@code true} tracks number of allocated pages (for tests purpose only). */ @@ -172,8 +172,8 @@ public class PageMemoryNoStoreImpl implements PageMemory { DirectMemoryProvider directMemoryProvider, GridCacheSharedContext sharedCtx, int pageSize, -MemoryPolicyConfiguration memPlcCfg, -MemoryMetricsImpl memMetrics, +DataRegionConfiguration dataRegionCfg, +DataRegionMetricsImpl memMetrics, boolean trackAcquiredPages ) { assert log != null || sharedCtx != null; @@ -183,21 +183,21 @@ public class PageMemoryNoStoreImpl implements PageMemory { this.directMemoryProvider = directMemoryProvider; this.trackAcquiredPages = trackAcquiredPages; this.memMetrics = memMetrics; -memoryPolicyCfg = memPlcCfg; +this.dataRegionCfg = dataRegionCfg; sysPageSize = pageSize + PAGE_OVERHEAD; assert sysPageSize % 8 == 0 : sysPageSize; -totalPages = (int)(memPlcCfg.getMaxSize() / sysPageSize); +totalPages = (int)(dataRegionCfg.getMaxSize() / sysPageSize); rwLock = new OffheapReadWriteLock(lockConcLvl); } /** {@inheritDoc} */ @Override public void start() throws IgniteException { -long startSize = memoryPolicyCfg.getInitialSize(); -long maxSize = memoryPolicyCfg.getMaxSize(); +long startSize = dataRegionCfg.getInitialSize(); +long maxSize = dataRegionCfg.getMaxSize(); long[] chunks = new long[SEG_CNT]; @@ -290,9 +290,9 @@ public class PageMemoryNoStoreImpl implements PageMemory { if (relPtr == INVALID_REL_PTR) throw new IgniteOutOfMemoryException("Not enough memory allocated " + -"(consider increasing memory policy size or enabling evictions) " + -"[policyName=" + memoryPolicyCfg.getName() + -", size=" + U.readableSize(memoryPolicyCfg.getMaxSize(), true) + "]" +"(consider increasing data region size or enabling evictions) " + +"[policyName=" + dataRegionCfg.getName() + +", size=" + U.readableSize(d
[05/11] ignite git commit: IGNITE-6030 Allow enabling persistence per data region
http://git-wip-us.apache.org/repos/asf/ignite/blob/ec41370c/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/GridMemoryConfigurationConsistencySelfTest.java -- diff --git a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/GridMemoryConfigurationConsistencySelfTest.java b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/GridMemoryConfigurationConsistencySelfTest.java deleted file mode 100644 index bc71e33..000 --- a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/GridMemoryConfigurationConsistencySelfTest.java +++ /dev/null @@ -1,79 +0,0 @@ -/* -* Licensed to the Apache Software Foundation (ASF) under one or more -* contributor license agreements. See the NOTICE file distributed with -* this work for additional information regarding copyright ownership. -* The ASF licenses this file to You under the Apache License, Version 2.0 -* (the "License"); you may not use this file except in compliance with -* the License. You may obtain a copy of the License at -* -* http://www.apache.org/licenses/LICENSE-2.0 -* -* Unless required by applicable law or agreed to in writing, software -* distributed under the License is distributed on an "AS IS" BASIS, -* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -* See the License for the specific language governing permissions and -* limitations under the License. -*/ - -package org.apache.ignite.internal.processors.cache; - -import java.util.concurrent.Callable; -import org.apache.ignite.IgniteCheckedException; -import org.apache.ignite.configuration.IgniteConfiguration; -import org.apache.ignite.configuration.MemoryConfiguration; -import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi; -import org.apache.ignite.spi.discovery.tcp.ipfinder.TcpDiscoveryIpFinder; -import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder; -import org.apache.ignite.testframework.GridTestUtils; -import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest; - -/** - * Tests a check of memory configuration consistency. - */ -public class GridMemoryConfigurationConsistencySelfTest extends GridCommonAbstractTest { -/** IP finder. */ -private static final TcpDiscoveryIpFinder IP_FINDER = new TcpDiscoveryVmIpFinder(true); - -/** {@inheritDoc} */ -@Override protected IgniteConfiguration getConfiguration(String gridName) throws Exception { -IgniteConfiguration cfg = super.getConfiguration(gridName); - -TcpDiscoverySpi discoSpi = new TcpDiscoverySpi(); -discoSpi.setIpFinder(IP_FINDER); - -cfg.setDiscoverySpi(discoSpi); - -MemoryConfiguration memCfg = new MemoryConfiguration(); - -// Nodes will have different page size. -memCfg.setPageSize(MemoryConfiguration.DFLT_PAGE_SIZE * (1 + getTestIgniteInstanceIndex(gridName))); - -cfg.setMemoryConfiguration(memCfg); - -return cfg; -} - -/** - * @throws Exception If failed. - */ -public void testMemoryConfigurationConsistency() throws Exception { -GridTestUtils.assertThrows(log, new Callable() { -/** {@inheritDoc} */ -@Override public Void call() throws Exception { -startGrids(2); - -return null; -} -}, IgniteCheckedException.class, null); -} - -/** {@inheritDoc} */ -@Override protected void beforeTest() throws Exception { -stopAllGrids(); -} - -/** {@inheritDoc} */ -@Override protected void afterTest() throws Exception { -stopAllGrids(); -} -} http://git-wip-us.apache.org/repos/asf/ignite/blob/ec41370c/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/IgniteClusterActivateDeactivateTest.java -- diff --git a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/IgniteClusterActivateDeactivateTest.java b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/IgniteClusterActivateDeactivateTest.java index 4c9ad27..1827c65 100644 --- a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/IgniteClusterActivateDeactivateTest.java +++ b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/IgniteClusterActivateDeactivateTest.java @@ -29,9 +29,9 @@ import org.apache.ignite.IgniteCache; import org.apache.ignite.cache.CacheAtomicityMode; import org.apache.ignite.cluster.ClusterNode; import org.apache.ignite.configuration.CacheConfiguration; +import org.apache.ignite.configuration.DataRegionConfiguration; +import org.apache.ignite.configuration.DataStorageConfiguration; import org.apache.ignite.configuration.IgniteConfiguration; -import org.apache.ignite.configuration.MemoryConfiguration; -import org.apache.ignite.configuration.PersistentStoreCon
[06/11] ignite git commit: IGNITE-6030 Allow enabling persistence per data region
http://git-wip-us.apache.org/repos/asf/ignite/blob/ec41370c/modules/core/src/main/java/org/apache/ignite/internal/visor/node/VisorDataRegionConfiguration.java -- diff --git a/modules/core/src/main/java/org/apache/ignite/internal/visor/node/VisorDataRegionConfiguration.java b/modules/core/src/main/java/org/apache/ignite/internal/visor/node/VisorDataRegionConfiguration.java new file mode 100644 index 000..394e294 --- /dev/null +++ b/modules/core/src/main/java/org/apache/ignite/internal/visor/node/VisorDataRegionConfiguration.java @@ -0,0 +1,225 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.ignite.internal.visor.node; + +import java.io.IOException; +import java.io.ObjectInput; +import java.io.ObjectOutput; +import java.util.ArrayList; +import java.util.List; +import org.apache.ignite.configuration.DataPageEvictionMode; +import org.apache.ignite.configuration.DataRegionConfiguration; +import org.apache.ignite.internal.util.typedef.internal.S; +import org.apache.ignite.internal.util.typedef.internal.U; +import org.apache.ignite.internal.visor.VisorDataTransferObject; + +/** + * Data transfer object for data region configuration. + */ +public class VisorDataRegionConfiguration extends VisorDataTransferObject { +/** */ +private static final long serialVersionUID = 0L; + +/** Unique name of DataRegion. */ +private String name; + +/** Initial memory region size defined by this memory policy. */ +private long initSize; + +/** Maximum memory region size defined by this memory policy. */ +private long maxSize; + +/** Path for memory mapped file. */ +private String swapPath; + +/** An algorithm for memory pages eviction. */ +private DataPageEvictionMode pageEvictionMode; + +/** A threshold for memory pages eviction initiation. */ +private double evictionThreshold; + +/** Minimum number of empty pages in reuse lists. */ +private int emptyPagesPoolSize; + +/** Enable memory metrics collection for this data region. */ +private boolean metricsEnabled; + +/** Number of sub-intervals. */ +private int metricsSubIntervalCount; + +/** Time interval over which allocation rate is calculated. */ +private long metricsRateTimeInterval; + +/** Enable Ignite Native Persistence. */ +private boolean persistenceEnabled; + +/** + * Default constructor. + */ +public VisorDataRegionConfiguration() { +// No-op. +} + +/** + * Constructor. + * + * @param plc Data region configuration. + */ +public VisorDataRegionConfiguration(DataRegionConfiguration plc) { +assert plc != null; + +name = plc.getName(); +initSize = plc.getInitialSize(); +maxSize = plc.getMaxSize(); +swapPath = plc.getSwapPath(); +pageEvictionMode = plc.getPageEvictionMode(); +evictionThreshold = plc.getEvictionThreshold(); +emptyPagesPoolSize = plc.getEmptyPagesPoolSize(); +metricsEnabled = plc.isMetricsEnabled(); +metricsSubIntervalCount = plc.getMetricsSubIntervalCount(); +metricsRateTimeInterval = plc.getMetricsRateTimeInterval(); +persistenceEnabled = plc.isPersistenceEnabled(); +} + +/** + * @param regCfgs Array of data region configurations. + * @return Collection of DTO objects. + */ +public static List from(DataRegionConfiguration[] regCfgs) { +List res = new ArrayList<>(); + +if (regCfgs != null) { +for (DataRegionConfiguration plc: regCfgs) +res.add(new VisorDataRegionConfiguration(plc)); +} + +return res; +} + +/** + * @return Unique name of DataRegion. + */ +public String getName() { +return name; +} + +/** + * @return Maximum memory region size defined by this memory policy. + */ +public long getMaxSize() { +return maxSize; +} + +/** + * @return Initial memory region size defined by this memory policy. + */ +public long getInitialSize() { +return initSize; +} + +/** + * @return Path for memor
[10/11] ignite git commit: IGNITE-6030 Allow enabling persistence per data region
http://git-wip-us.apache.org/repos/asf/ignite/blob/ec41370c/modules/core/src/main/java/org/apache/ignite/configuration/DataRegionConfiguration.java -- diff --git a/modules/core/src/main/java/org/apache/ignite/configuration/DataRegionConfiguration.java b/modules/core/src/main/java/org/apache/ignite/configuration/DataRegionConfiguration.java new file mode 100644 index 000..50edf5c --- /dev/null +++ b/modules/core/src/main/java/org/apache/ignite/configuration/DataRegionConfiguration.java @@ -0,0 +1,406 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.ignite.configuration; + +import java.io.Serializable; +import org.apache.ignite.DataRegionMetrics; +import org.apache.ignite.internal.mem.IgniteOutOfMemoryException; +import org.apache.ignite.mxbean.DataRegionMetricsMXBean; + +import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_DATA_REG_DEFAULT_NAME; + +/** + * This class allows defining custom data regions' configurations with various parameters for Apache Ignite + * page memory (see {@link DataStorageConfiguration}. For each configured data region Apache Ignite instantiates + * respective memory regions with different parameters like maximum size, eviction policy, swapping options, + * persistent mode flag, etc. + * An Apache Ignite cache can be mapped to a particular region using + * {@link CacheConfiguration#setDataRegionName(String)} method. + * Sample configuration below shows how to configure several data regions: + * + * {@code + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * } + * + */ +public final class DataRegionConfiguration implements Serializable { +/** */ +private static final long serialVersionUID = 0L; + +/** Default metrics enabled flag. */ +public static final boolean DFLT_METRICS_ENABLED = false; + +/** Default amount of sub intervals to calculate {@link DataRegionMetrics#getAllocationRate()} metric. */ +public static final int DFLT_SUB_INTERVALS = 5; + +/** Default length of interval over which {@link DataRegionMetrics#getAllocationRate()} metric is calculated. */ +public static final int DFLT_RATE_TIME_INTERVAL_MILLIS = 60_000; + +/** Data region name. */ +private String name = DFLT_DATA_REG_DEFAULT_NAME; + +/** Data region maximum size in memory. */ +private long maxSize = DataStorageConfiguration.DFLT_DATA_REGION_MAX_SIZE; + +/** Data region start size. */ +private long initSize = Math.min( +DataStorageConfiguration.DFLT_DATA_REGION_MAX_SIZE, DataStorageConfiguration.DFLT_DATA_REGION_INITIAL_SIZE); + +/** An optional path to a memory mapped files directory for this data region. */ +private String swapPath; + +/** An algorithm for memory pages eviction. */ +private DataPageEvictionMode pageEvictionMode = DataPageEvictionMode.DISABLED; + +/** + * A threshold for memory pages eviction initiation. For instance, if the threshold is 0.9 it means that the page + * memory will start the eviction only after 90% data region is occupied. + */ +private double evictionThreshold = 0.9; + +/** Minimum number of empty pages in reuse lists. */ +private int emptyPagesPoolSize = 100; + +/** + * Flag to enable the memory metrics collection for this data region. + */ +private boolean metricsEnabled = DFLT_METRICS_ENABLED; + +/** Number of sub-intervals the whole {@link #setMetricsRateTimeInterval(long)} will be split into to calculate + * {@link DataRegionMetrics#getAllocationRate()} and {@link DataRegionMetrics#getEvictionRate()} rates (5 by
[01/11] ignite git commit: IGNITE-6030 Allow enabling persistence per data region
Repository: ignite Updated Branches: refs/heads/master 173ecef0f -> ec41370c4 http://git-wip-us.apache.org/repos/asf/ignite/blob/ec41370c/modules/web-console/frontend/app/modules/states/configuration/clusters/data-storage.pug -- diff --git a/modules/web-console/frontend/app/modules/states/configuration/clusters/data-storage.pug b/modules/web-console/frontend/app/modules/states/configuration/clusters/data-storage.pug new file mode 100644 index 000..9c2dca1 --- /dev/null +++ b/modules/web-console/frontend/app/modules/states/configuration/clusters/data-storage.pug @@ -0,0 +1,255 @@ +//- +Licensed to the Apache Software Foundation (ASF) under one or more +contributor license agreements. See the NOTICE file distributed with +this work for additional information regarding copyright ownership. +The ASF licenses this file to You under the Apache License, Version 2.0 +(the "License"); you may not use this file except in compliance with +the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +include /app/helpers/jade/mixins + +-var form = 'dataStorageConfiguration' +-var model = 'backupItem.dataStorageConfiguration' +-var dfltRegionModel = model + '.defaultDataRegionConfiguration' +-var dataRegionConfigurations = model + '.dataRegionConfigurations' + +.panel.panel-default(ng-show='$ctrl.available("2.3.0")' ng-form=form novalidate) +.panel-heading(bs-collapse-toggle ng-click=`ui.loadPanel('${form}')`) +ignite-form-panel-chevron +label Data storage configuration +ignite-form-field-tooltip.tipLabel +| Page memory is a manageable off-heap based memory architecture that is split into pages of fixed size#[br] +| #[a(href="https://apacheignite.readme.io/docs/distributed-persistent-store"; target="_blank") More info] +ignite-form-revert +.panel-collapse(role='tabpanel' bs-collapse-target id=`${form}`) +.panel-body(ng-if=`$ctrl.available("2.3.0") && ui.isPanelLoaded('${form}')`) +.col-sm-6 +.settings-row ++number-min-max('Page size:', model + '.pageSize', '"DataStorageConfigurationPageSize"', +'true', '2048', '1024', '16384', 'Every memory region is split on pages of fixed size') +.settings-row ++number('Concurrency level:', model + '.concurrencyLevel', '"DataStorageConfigurationConcurrencyLevel"', +'true', 'availableProcessors', '2', 'The number of concurrent segments in Ignite internal page mapping tables') +.settings-row ++ignite-form-group +ignite-form-field-label +| System region +ignite-form-group-tooltip +| System region properties +.group-content +.details-row ++number('Initial size:', model + '.systemRegionInitialSize', '"DataStorageSystemRegionInitialSize"', +'true', '41943040', '10485760', 'Initial size of a data region reserved for system cache') +.details-row ++number('Maximum size:', model + '.systemRegionMaxSize', '"DataStorageSystemRegionMaxSize"', +'true', '104857600', '10485760', 'Maximum data region size reserved for system cache') +.settings-row ++ignite-form-group +ignite-form-field-label +| Data regions +ignite-form-group-tooltip +| Data region configurations +.group-content +.details-row ++ignite-form-group +ignite-form-field-label +| Default data region +ignite-form-group-tooltip +| Default data region properties +.group-content +.details-row ++text('Name:', dfltRegionModel + '.name', '"DfltRegionName" + $index', 'false', 'default', 'Default data region name') +.details-row ++number('Initial size:',
[08/11] ignite git commit: IGNITE-6030 Allow enabling persistence per data region
http://git-wip-us.apache.org/repos/asf/ignite/blob/ec41370c/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheDatabaseSharedManager.java -- diff --git a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheDatabaseSharedManager.java b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheDatabaseSharedManager.java index 9a2e028..de3b60a 100755 --- a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheDatabaseSharedManager.java +++ b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheDatabaseSharedManager.java @@ -57,18 +57,17 @@ import java.util.concurrent.locks.ReentrantReadWriteLock; import java.util.regex.Matcher; import java.util.regex.Pattern; import javax.management.ObjectName; +import org.apache.ignite.DataStorageMetrics; import org.apache.ignite.IgniteCheckedException; import org.apache.ignite.IgniteException; import org.apache.ignite.IgniteLogger; import org.apache.ignite.IgniteSystemProperties; -import org.apache.ignite.PersistenceMetrics; import org.apache.ignite.cluster.ClusterNode; import org.apache.ignite.configuration.CheckpointWriteOrder; import org.apache.ignite.configuration.DataPageEvictionMode; +import org.apache.ignite.configuration.DataRegionConfiguration; +import org.apache.ignite.configuration.DataStorageConfiguration; import org.apache.ignite.configuration.IgniteConfiguration; -import org.apache.ignite.configuration.MemoryConfiguration; -import org.apache.ignite.configuration.MemoryPolicyConfiguration; -import org.apache.ignite.configuration.PersistentStoreConfiguration; import org.apache.ignite.events.DiscoveryEvent; import org.apache.ignite.events.EventType; import org.apache.ignite.internal.GridKernalContext; @@ -139,7 +138,7 @@ import org.apache.ignite.internal.util.typedef.internal.U; import org.apache.ignite.internal.util.worker.GridWorker; import org.apache.ignite.lang.IgniteBiTuple; import org.apache.ignite.lang.IgniteOutClosure; -import org.apache.ignite.mxbean.PersistenceMetricsMXBean; +import org.apache.ignite.mxbean.DataStorageMetricsMXBean; import org.apache.ignite.thread.IgniteThread; import org.apache.ignite.thread.IgniteThreadPoolExecutor; import org.jetbrains.annotations.NotNull; @@ -229,7 +228,7 @@ public class GridCacheDatabaseSharedManager extends IgniteCacheDatabaseSharedMan }; /** */ -private static final String MBEAN_NAME = "PersistenceMetrics"; +private static final String MBEAN_NAME = "DataStorageMetrics"; /** */ private static final String MBEAN_GROUP = "Persistent Store"; @@ -262,7 +261,7 @@ public class GridCacheDatabaseSharedManager extends IgniteCacheDatabaseSharedMan private volatile boolean printCheckpointStats = true; /** Database configuration. */ -private final PersistentStoreConfiguration persistenceCfg; +private final DataStorageConfiguration persistenceCfg; /** */ private final Collection lsnrs = new CopyOnWriteArrayList<>(); @@ -301,7 +300,7 @@ public class GridCacheDatabaseSharedManager extends IgniteCacheDatabaseSharedMan private IgniteCacheSnapshotManager snapshotMgr; /** */ -private PersistenceMetricsImpl persStoreMetrics; +private DataStorageMetricsImpl persStoreMetrics; /** */ private ObjectName persistenceMetricsMbeanName; @@ -318,18 +317,18 @@ public class GridCacheDatabaseSharedManager extends IgniteCacheDatabaseSharedMan public GridCacheDatabaseSharedManager(GridKernalContext ctx) { IgniteConfiguration cfg = ctx.config(); -persistenceCfg = cfg.getPersistentStoreConfiguration(); +persistenceCfg = cfg.getDataStorageConfiguration(); -assert persistenceCfg != null : "PageStore should not be created if persistence is disabled."; +assert persistenceCfg != null; -checkpointFreq = persistenceCfg.getCheckpointingFrequency(); +checkpointFreq = persistenceCfg.getCheckpointFrequency(); lockWaitTime = persistenceCfg.getLockWaitTime(); -persStoreMetrics = new PersistenceMetricsImpl( +persStoreMetrics = new DataStorageMetricsImpl( persistenceCfg.isMetricsEnabled(), -persistenceCfg.getRateTimeInterval(), -persistenceCfg.getSubIntervals() +persistenceCfg.getMetricsRateTimeInterval(), +persistenceCfg.getMetricsSubIntervalCount() ); } @@ -400,12 +399,12 @@ public class GridCacheDatabaseSharedManager extends IgniteCacheDatabaseSharedMan * */ private void initDataBase() { -if (persistenceCfg.getCheckpointingThreads() > 1) +if (persistenceCfg.getCheckpointThreads() > 1) asyncRunner = new IgniteThreadPoolExecutor( "checkpoint-runner",
[07/11] ignite git commit: IGNITE-6030 Allow enabling persistence per data region
http://git-wip-us.apache.org/repos/asf/ignite/blob/ec41370c/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/MemoryMetricsSnapshot.java -- diff --git a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/MemoryMetricsSnapshot.java b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/MemoryMetricsSnapshot.java deleted file mode 100644 index 4e7f90a..000 --- a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/MemoryMetricsSnapshot.java +++ /dev/null @@ -1,112 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.ignite.internal.processors.cache.persistence; - -import org.apache.ignite.MemoryMetrics; - -/** - * - */ -public class MemoryMetricsSnapshot implements MemoryMetrics { -/** */ -private String name; - -/** */ -private long totalAllocatedPages; - -/** */ -private float allocationRate; - -/** */ -private float evictionRate; - -/** */ -private float largeEntriesPagesPercentage; - -/** */ -private float pagesFillFactor; - -/** */ -private long dirtyPages; - -/** */ -private float pageReplaceRate; - -/** */ -private long physicalMemoryPages; - -/** - * @param metrics Metrics instance to take a copy. - */ -public MemoryMetricsSnapshot(MemoryMetrics metrics) { -name = metrics.getName(); -totalAllocatedPages = metrics.getTotalAllocatedPages(); -allocationRate = metrics.getAllocationRate(); -evictionRate = metrics.getEvictionRate(); -largeEntriesPagesPercentage = metrics.getLargeEntriesPagesPercentage(); -pagesFillFactor = metrics.getPagesFillFactor(); -dirtyPages = metrics.getDirtyPages(); -pageReplaceRate = metrics.getPagesReplaceRate(); -physicalMemoryPages = metrics.getPhysicalMemoryPages(); -} - -/** {@inheritDoc} */ -@Override public String getName() { -return name; -} - -/** {@inheritDoc} */ -@Override public long getTotalAllocatedPages() { -return totalAllocatedPages; -} - -/** {@inheritDoc} */ -@Override public float getAllocationRate() { -return allocationRate; -} - -/** {@inheritDoc} */ -@Override public float getEvictionRate() { -return evictionRate; -} - -/** {@inheritDoc} */ -@Override public float getLargeEntriesPagesPercentage() { -return largeEntriesPagesPercentage; -} - -/** {@inheritDoc} */ -@Override public float getPagesFillFactor() { -return pagesFillFactor; -} - -/** {@inheritDoc} */ -@Override public long getDirtyPages() { -return dirtyPages; -} - -/** {@inheritDoc} */ -@Override public float getPagesReplaceRate() { -return pageReplaceRate; -} - -/** {@inheritDoc} */ -@Override public long getPhysicalMemoryPages() { -return physicalMemoryPages; -} -} http://git-wip-us.apache.org/repos/asf/ignite/blob/ec41370c/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/MemoryPolicy.java -- diff --git a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/MemoryPolicy.java b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/MemoryPolicy.java deleted file mode 100644 index 4059c12..000 --- a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/MemoryPolicy.java +++ /dev/null @@ -1,84 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless
[02/11] ignite git commit: IGNITE-6030 Allow enabling persistence per data region
http://git-wip-us.apache.org/repos/asf/ignite/blob/ec41370c/modules/core/src/test/java/org/apache/ignite/internal/processors/database/MetadataStorageSelfTest.java -- diff --git a/modules/core/src/test/java/org/apache/ignite/internal/processors/database/MetadataStorageSelfTest.java b/modules/core/src/test/java/org/apache/ignite/internal/processors/database/MetadataStorageSelfTest.java index dcd4ce1..880e37e 100644 --- a/modules/core/src/test/java/org/apache/ignite/internal/processors/database/MetadataStorageSelfTest.java +++ b/modules/core/src/test/java/org/apache/ignite/internal/processors/database/MetadataStorageSelfTest.java @@ -23,12 +23,12 @@ import java.util.Map; import java.util.Random; import java.util.concurrent.ThreadLocalRandom; import java.util.concurrent.atomic.AtomicLong; -import org.apache.ignite.configuration.MemoryPolicyConfiguration; +import org.apache.ignite.configuration.DataRegionConfiguration; import org.apache.ignite.internal.mem.DirectMemoryProvider; import org.apache.ignite.internal.pagemem.FullPageId; import org.apache.ignite.internal.pagemem.PageIdAllocator; import org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl; -import org.apache.ignite.internal.processors.cache.persistence.MemoryMetricsImpl; +import org.apache.ignite.internal.processors.cache.persistence.DataRegionMetricsImpl; import org.apache.ignite.internal.processors.cache.persistence.MetadataStorage; import org.apache.ignite.internal.mem.file.MappedFileMemoryProvider; import org.apache.ignite.internal.pagemem.PageMemory; @@ -156,7 +156,7 @@ public class MetadataStorageSelfTest extends GridCommonAbstractTest { protected PageMemory memory(boolean clean) throws Exception { DirectMemoryProvider provider = new MappedFileMemoryProvider(log(), allocationPath); -MemoryPolicyConfiguration plcCfg = new MemoryPolicyConfiguration() +DataRegionConfiguration plcCfg = new DataRegionConfiguration() .setMaxSize(30 * 1024 * 1024).setInitialSize(30 * 1024 * 1024); return new PageMemoryNoStoreImpl( @@ -165,7 +165,7 @@ public class MetadataStorageSelfTest extends GridCommonAbstractTest { null, PAGE_SIZE, plcCfg, -new MemoryMetricsImpl(plcCfg), +new DataRegionMetricsImpl(plcCfg), true); } } http://git-wip-us.apache.org/repos/asf/ignite/blob/ec41370c/modules/core/src/test/java/org/apache/ignite/internal/processors/database/SwapPathConstructionSelfTest.java -- diff --git a/modules/core/src/test/java/org/apache/ignite/internal/processors/database/SwapPathConstructionSelfTest.java b/modules/core/src/test/java/org/apache/ignite/internal/processors/database/SwapPathConstructionSelfTest.java index 53e5daf..f22128c 100644 --- a/modules/core/src/test/java/org/apache/ignite/internal/processors/database/SwapPathConstructionSelfTest.java +++ b/modules/core/src/test/java/org/apache/ignite/internal/processors/database/SwapPathConstructionSelfTest.java @@ -22,23 +22,23 @@ import java.nio.file.Path; import java.nio.file.Paths; import java.util.Map; import org.apache.ignite.configuration.IgniteConfiguration; -import org.apache.ignite.configuration.MemoryConfiguration; -import org.apache.ignite.configuration.MemoryPolicyConfiguration; +import org.apache.ignite.configuration.DataStorageConfiguration; +import org.apache.ignite.configuration.DataRegionConfiguration; import org.apache.ignite.internal.GridKernalContext; import org.apache.ignite.internal.IgniteEx; import org.apache.ignite.internal.pagemem.PageMemory; import org.apache.ignite.internal.processors.cache.persistence.IgniteCacheDatabaseSharedManager; -import org.apache.ignite.internal.processors.cache.persistence.MemoryPolicy; +import org.apache.ignite.internal.processors.cache.persistence.DataRegion; import org.apache.ignite.internal.util.typedef.internal.U; import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest; /** - * Test verifies correct construction of swap file path {@link MemoryPolicyConfiguration#setSwapFilePath(String)} + * Test verifies correct construction of swap file path {@link DataRegionConfiguration#setSwapPath(String)} * when absolute or relative paths are provided via configuration. */ public class SwapPathConstructionSelfTest extends GridCommonAbstractTest { /** */ -private MemoryConfiguration memCfg; +private DataStorageConfiguration memCfg; /** */ private static final String RELATIVE_SWAP_PATH = "relSwapPath"; @@ -50,7 +50,7 @@ public class SwapPathConstructionSelfTest extends GridCommonAbstractTest { @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception { IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName); -cfg.setMemory
[04/11] ignite git commit: IGNITE-6030 Allow enabling persistence per data region
http://git-wip-us.apache.org/repos/asf/ignite/blob/ec41370c/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/MemoryPolicyInitializationTest.java -- diff --git a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/MemoryPolicyInitializationTest.java b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/MemoryPolicyInitializationTest.java index 842f618..fb1574d 100644 --- a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/MemoryPolicyInitializationTest.java +++ b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/MemoryPolicyInitializationTest.java @@ -73,7 +73,7 @@ public class MemoryPolicyInitializationTest extends GridCommonAbstractTest { IgniteEx ignite = startGrid(0); -Collection allMemPlcs = ignite.context().cache().context().database().memoryPolicies(); +Collection allMemPlcs = ignite.context().cache().context().database().dataRegions(); assertTrue(allMemPlcs.size() == 2); @@ -89,7 +89,7 @@ public class MemoryPolicyInitializationTest extends GridCommonAbstractTest { IgniteEx ignite = startGrid(0); -Collection allMemPlcs = ignite.context().cache().context().database().memoryPolicies(); +Collection allMemPlcs = ignite.context().cache().context().database().dataRegions(); assertTrue(allMemPlcs.size() == 3); @@ -110,13 +110,13 @@ public class MemoryPolicyInitializationTest extends GridCommonAbstractTest { IgniteCacheDatabaseSharedManager dbMgr = ignite.context().cache().context().database(); -Collection allMemPlcs = dbMgr.memoryPolicies(); +Collection allMemPlcs = dbMgr.dataRegions(); assertTrue(allMemPlcs.size() == 2); verifyDefaultAndSystemMemoryPolicies(allMemPlcs); -MemoryPolicy dfltMemPlc = U.field(dbMgr, "dfltMemPlc"); +DataRegion dfltMemPlc = U.field(dbMgr, "dfltDataRegion"); assertTrue(dfltMemPlc.config().getMaxSize() == USER_DEFAULT_MEM_PLC_SIZE); } @@ -134,13 +134,13 @@ public class MemoryPolicyInitializationTest extends GridCommonAbstractTest { IgniteCacheDatabaseSharedManager dbMgr = ignite.context().cache().context().database(); -Collection allMemPlcs = dbMgr.memoryPolicies(); +Collection allMemPlcs = dbMgr.dataRegions(); assertTrue(allMemPlcs.size() == 3); verifyDefaultAndSystemMemoryPolicies(allMemPlcs); -MemoryPolicy dfltMemPlc = U.field(dbMgr, "dfltMemPlc"); +DataRegion dfltMemPlc = U.field(dbMgr, "dfltDataRegion"); assertTrue(dfltMemPlc.config().getMaxSize() == USER_CUSTOM_MEM_PLC_SIZE); } @@ -220,7 +220,7 @@ public class MemoryPolicyInitializationTest extends GridCommonAbstractTest { private void verifyCacheMemoryPolicy(IgniteCache cache, String plcName) { GridCacheContext ctx = ((IgniteCacheProxy) cache).context(); -assertEquals(plcName, ctx.memoryPolicy().config().getName()); +assertEquals(plcName, ctx.dataRegion().config().getName()); } /** @@ -278,12 +278,12 @@ public class MemoryPolicyInitializationTest extends GridCommonAbstractTest { /** * @param allMemPlcs Collection of all memory policies. */ -private void verifyDefaultAndSystemMemoryPolicies(Collection allMemPlcs) { +private void verifyDefaultAndSystemMemoryPolicies(Collection allMemPlcs) { assertTrue("Default memory policy is not presented", isMemoryPolicyPresented(allMemPlcs, DFLT_MEM_PLC_DEFAULT_NAME)); assertTrue("System memory policy is not presented", -isMemoryPolicyPresented(allMemPlcs, IgniteCacheDatabaseSharedManager.SYSTEM_MEMORY_POLICY_NAME)); +isMemoryPolicyPresented(allMemPlcs, IgniteCacheDatabaseSharedManager.SYSTEM_DATA_REGION_NAME)); } /** @@ -303,8 +303,8 @@ public class MemoryPolicyInitializationTest extends GridCommonAbstractTest { * @param memPlcs Collection of memory policies. * @param nameToVerify Excepted name of memory policy. */ -private boolean isMemoryPolicyPresented(Collection memPlcs, String nameToVerify) { -for (MemoryPolicy memPlc : memPlcs) { +private boolean isMemoryPolicyPresented(Collection memPlcs, String nameToVerify) { +for (DataRegion memPlc : memPlcs) { if (nameToVerify.equals(memPlc.config().getName())) return true; } http://git-wip-us.apache.org/repos/asf/ignite/blob/ec41370c/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsCacheRestoreTest.java -- diff --git a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/d
[11/11] ignite git commit: IGNITE-6030 Allow enabling persistence per data region
IGNITE-6030 Allow enabling persistence per data region Project: http://git-wip-us.apache.org/repos/asf/ignite/repo Commit: http://git-wip-us.apache.org/repos/asf/ignite/commit/ec41370c Tree: http://git-wip-us.apache.org/repos/asf/ignite/tree/ec41370c Diff: http://git-wip-us.apache.org/repos/asf/ignite/diff/ec41370c Branch: refs/heads/master Commit: ec41370c4da65c01e9b7c584e0706c40e4171b35 Parents: 173ecef Author: Ivan Rakov Authored: Fri Oct 20 10:29:57 2017 +0300 Committer: Alexey Goncharuk Committed: Fri Oct 20 10:29:57 2017 +0300 -- examples/config/example-data-regions.xml| 106 +++ examples/config/example-memory-policies.xml | 108 --- .../example-persistent-store.xml| 14 +- .../examples/datagrid/DataRegionsExample.java | 113 +++ .../datagrid/MemoryPoliciesExample.java | 114 --- .../ignite/examples/CacheExamplesSelfTest.java | 6 +- .../benchmarks/jmh/tree/BPlusTreeBenchmark.java | 8 +- .../DummyPersistenceCompatibilityTest.java | 8 +- .../org/apache/ignite/DataRegionMetrics.java| 119 +++ .../apache/ignite/DataRegionMetricsAdapter.java | 106 +++ .../org/apache/ignite/DataStorageMetrics.java | 114 +++ .../ignite/DataStorageMetricsAdapter.java | 101 +++ .../src/main/java/org/apache/ignite/Ignite.java | 48 +- .../apache/ignite/IgniteSystemProperties.java | 4 +- .../java/org/apache/ignite/MemoryMetrics.java | 11 +- .../org/apache/ignite/PersistenceMetrics.java | 4 +- .../configuration/CacheConfiguration.java | 36 +- .../configuration/DataPageEvictionMode.java | 8 +- .../configuration/DataRegionConfiguration.java | 406 + .../configuration/DataStorageConfiguration.java | 882 +++ .../configuration/IgniteConfiguration.java | 46 +- .../configuration/MemoryConfiguration.java | 9 +- .../MemoryPolicyConfiguration.java | 32 +- .../PersistentStoreConfiguration.java | 4 +- .../org/apache/ignite/igfs/IgfsMetrics.java | 4 +- .../apache/ignite/internal/IgniteKernal.java| 77 +- .../ignite/internal/IgniteNodeAttributes.java | 4 + .../org/apache/ignite/internal/IgnitionEx.java | 132 ++- .../ignite/internal/MarshallerContextImpl.java | 3 +- .../discovery/GridDiscoveryManager.java | 31 +- .../pagemem/impl/PageMemoryNoStoreImpl.java | 32 +- .../cache/CacheAffinitySharedManager.java | 4 +- .../processors/cache/CacheGroupContext.java | 29 +- .../processors/cache/CacheGroupData.java| 17 +- .../processors/cache/CacheGroupDescriptor.java | 19 +- .../processors/cache/ClusterCachesInfo.java | 13 +- .../processors/cache/GridCacheAdapter.java | 3 +- .../processors/cache/GridCacheContext.java | 8 +- .../processors/cache/GridCacheMapEntry.java | 10 +- .../processors/cache/GridCacheProcessor.java| 55 +- .../processors/cache/GridCacheUtils.java| 55 ++ .../cache/IgniteCacheOffheapManagerImpl.java| 6 +- .../cache/binary/BinaryMetadataFileStore.java | 7 +- .../GridDistributedTxRemoteAdapter.java | 5 +- .../distributed/dht/GridDhtLocalPartition.java | 2 +- .../dht/atomic/GridDhtAtomicCache.java | 3 +- .../distributed/near/GridNearGetRequest.java| 2 +- .../distributed/near/GridNearLockRequest.java | 2 +- .../cache/distributed/near/GridNearTxLocal.java | 2 +- .../near/GridNearTxPrepareRequest.java | 2 +- .../local/atomic/GridLocalAtomicCache.java | 2 +- .../cache/persistence/CacheDataRowAdapter.java | 2 +- .../cache/persistence/DataRegion.java | 84 ++ .../persistence/DataRegionMetricsImpl.java | 286 ++ .../DataRegionMetricsMXBeanImpl.java| 131 +++ .../persistence/DataRegionMetricsSnapshot.java | 112 +++ .../persistence/DataStorageMetricsImpl.java | 297 +++ .../persistence/DataStorageMetricsSnapshot.java | 144 +++ .../GridCacheDatabaseSharedManager.java | 166 ++-- .../persistence/GridCacheOffheapManager.java| 22 +- .../IgniteCacheDatabaseSharedManager.java | 538 +-- .../cache/persistence/MemoryMetricsImpl.java| 286 -- .../persistence/MemoryMetricsMXBeanImpl.java| 131 --- .../persistence/MemoryMetricsSnapshot.java | 112 --- .../cache/persistence/MemoryPolicy.java | 84 -- .../persistence/PersistenceMetricsImpl.java | 297 --- .../persistence/PersistenceMetricsSnapshot.java | 144 --- .../processors/cache/persistence/RowStore.java | 38 +- .../evict/FairFifoPageEvictionTracker.java | 6 +- .../evict/PageAbstractEvictionTracker.java | 6 +- .../evict/Random2LruPageEvictionTracker.java| 8 +- .../evict/RandomLruPageEvictionTracker.java | 8 +- .../persistence/file/AsyncFileIOFactory.java| 28 +- .../cache/persistence/file/FilePageStore.java | 6 +- .../persistence/file/Fi