Author: dmagda Date: Tue Feb 11 00:45:25 2020 New Revision: 1873876 URL: http://svn.apache.org/viewvc?rev=1873876&view=rev Log: update memory-centric storage page
Modified: ignite/site/branches/ignite-redisign/arch/memorycentric.html ignite/site/branches/ignite-redisign/includes/header.html ignite/site/branches/ignite-redisign/use-cases/hadoop-acceleration.html ignite/site/branches/ignite-redisign/use-cases/in-memory-cache.html ignite/site/branches/ignite-redisign/use-cases/in-memory-database.html ignite/site/branches/ignite-redisign/use-cases/spark-acceleration.html Modified: ignite/site/branches/ignite-redisign/arch/memorycentric.html URL: http://svn.apache.org/viewvc/ignite/site/branches/ignite-redisign/arch/memorycentric.html?rev=1873876&r1=1873875&r2=1873876&view=diff ============================================================================== --- ignite/site/branches/ignite-redisign/arch/memorycentric.html (original) +++ ignite/site/branches/ignite-redisign/arch/memorycentric.html Tue Feb 11 00:45:25 2020 @@ -33,13 +33,19 @@ under the License. <!DOCTYPE html> <html lang="en"> <head> - <link rel="canonical" href="https://ignite.apache.org/arch/memorycentric.html" /> + <link rel="canonical" href="https://ignite.apache.org/arch/memorycentric.html"/> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> - <meta http-equiv="Cache-Control" content="no-cache, no-store, must-revalidate" /> - <meta http-equiv="Pragma" content="no-cache" /> - <meta http-equiv="Expires" content="0" /> - <title>Memory Centric - Apache Ignite</title> + <meta http-equiv="Cache-Control" content="no-cache, no-store, must-revalidate"/> + <meta http-equiv="Pragma" content="no-cache"/> + <meta http-equiv="Expires" content="0"/> + + <title>Memory-Centric Storage - Apache Ignite</title> + + <meta name="description" + content="Apache Ignite memory-centric storage uses both memory and disk as active storage tiers. Speed of + memory with the consistency of disk-based databases and no need for memory warm-ups on restarts."/> + <!--#include virtual="/includes/styles.html" --> <!--#include virtual="/includes/sh.html" --> @@ -54,13 +60,11 @@ under the License. <div class="col-sm-12 col-md-12 col-xs-12" style="padding:0 0 20px 0;"> <div class="col-sm-6 col-md-6 col-xs-12" style="padding-left:0; padding-right:0"> <p> - Apache Ignite is based on distributed <i>memory-centric architecture</i> that combines the - performance and scale of in-memory computing together with the disk durability and strong - consistency in one system. - </p> - <p> - When native persistence is turned on, Ignite functions as a <nobr><i>memory-centric system-of-record</i></nobr>, - where most of the processing happens in memory on cached data, but the superset of data and indexes gets persisted to disk. + Apache Ignite is designed to work with both memory and disk as active storage tiers. + Ignite storage is classified as a memory-centric one since most of the architectural decisions + favor specificities of the memory tier that is always at use and cannot be disabled. + The disk tier is optional with the support of two options -- you can persist data in an + external database or keep it in the Ignite native persistence. </p> </div> <div class="col-sm-6 col-md-6 col-xs-12" style="padding-right:0"> @@ -68,28 +72,27 @@ under the License. </div> </div> <p> - When persistence is turned off, Ignite functions as a memory-only store, in which case it can be treated as a Distributed Cache, - In-Memory Database (IMDB) or In-Memory Data Grid (IMDG). - </p> - - <div class="page-heading" id="db-and-caching-in-one"> - Database and Caching in One - <a href="/arch/memorycentric.html#db-and-caching-in-one"><i class="fa fa-anchor"></i></a> + Ignite takes full control of its memory tier by allocating and managing off-heap regions. Each Ignite + server node allocates memory regions during bootstrap, splits the regions into pages, and keeps data + records with indexes there. Java heap is used to keep temporary objects such as queries results sets, + metrics samples, objects generated by your application code. All these objects are garbage collected + eventually. + </p> + <p> + If you select the native persistence as the disk tier, then most of the processing will still + take place in memory on cached data, but, with a full copy stored on disk. If any record is + missing in memory, Ignite will read it from disk, allowing you to persist much larger data sets + than you can cache in memory. That also eliminates the need for time-consuming memory warm-ups + on restarts. As soon as your cluster reconnects after a restart, Ignite will serve most of the + data from disk warming the memory tier up in the background. + </p> + + <div class="page-heading" id="storage-usage-modes"> + Memory-Centric Storage Usage Modes + <a href="/arch/memorycentric.html#storage-usage-modes"><i class="fa fa-anchor"></i></a> </div> <p> - One of the main advantages of Ignite is that it comes with a distributed in-memory cache and a - distributed on-disk storage in one platform. In other words, Ignite users get both, a - distributed cache and a distributed database together. - </p> - <p> - In partitioned (not replicated) mode, the data is partitioned across multiple servers, with each server responsible - for a subset of the data. Collectively, the full data set is stored across all servers. Each server - has its subset persisted on disk. Depending on how much memory is available, - each server also has either the whole subset or a portion of it cached in memory. Such combination - of memory and disk creates a distributed memory-centric storage. - </p> - <p> - The following <i>memory and disk usage modes</i> are supported: + Below you can see primary modes for memory-centric storage usage and configuration: </p> <table class="formatted" name="Deployment Options Features"> <thead> @@ -103,268 +106,120 @@ under the License. <td class="left">In-Memory</td> <td> <p> - The whole data set is stored in memory. In order to survive - node failures, it is recommended to configure a number of redundant backup - copies (aka. replication factor) across the cluster. - </p> - - <p> - <strong>Use cases</strong>: in-memory caches, in-memory data grids, in-memory computations, - web-session caching, real-time processing of continuous data streams. - </p> - </td> - </tr> - <tr> - <td class="left">In-Memory + 3rd party database</td> - <td> - <p> - Ignite can be used as a caching layer (aka. data grid) over any existing 3rd party - database. This mode is used to accelerate and scale the underlying database. - Automatic integration is provided with most known databases, like Oracle, MySQL, - PostgreSQL, Apache Cassandra, etc. + The whole data set is available in memory only. In order to survive node failures, it is + recommended to keep at least one backup copy of the data across the cluster. </p> <p> - <strong>Use cases</strong>: Ignite as <a href="/features/datagrid.html">In-Memory Data Grid</a> - - adds acceleration and scale to existing database deployments (RDBMS, NoSQL, etc). + <strong>Use cases</strong>: general in-memory caching, high-performance + computing, web-session caching, real-time processing of continuous data streams. </p> </td> </tr> <tr> - <td class="left">In-Memory + Full Copy on Disk</td> + <td class="left">In-Memory + External Database</td> <td> <p> - The whole data set is stored in memory and on disk. The disk is used as a memory-offload for data recovery - purposes, in case of full cluster crashes and restarts. + Ignite is deployed as a distributed caching layer on top of an existing external database. + This mode is for the acceleration of disk-based databases and your services with APIs that + interact with them. </p> <p> - <strong>Use cases</strong>: Ignite as an - <a href="/use-cases/database/in-memory-database.html"><nobr>In-Memory Database</nobr></a> - - provides SQL, key-value and collocated processing APIs over in-memory data. + <strong>Use cases</strong>: acceleration of services and APIs with write-through/behind + capability, to an external database (aka. in-memory data grid). </p> </td> </tr> <tr> - <td class="left">100% on Disk + In-Memory Cache</td> + <td class="left">In-Memory Cache + Native Persistence</td> <td> <p> - 100% of data is persisted to disk and the same or smaller amount is cached in memory. - The more data is cached, the faster is the performance. The disk serves as the primary storage that survives - any type of cluster failures and restarts. + 100% of data is persisted to disk, and the same or smaller amount is cached in memory. + The more data is cached, the faster is the performance. The disk serves as the primary + storage that survives any cluster failures and restarts. No need for memory warm-ups on + restarts as long as Ignite can serve data from disk. </p> <p> - <strong>Use cases</strong>: Ignite as a - <a href="/use-cases/database/distributed-database.html">Memory-Centric Distributed Database</a> - - provides distributed database with SQL, key-value and collocated processing APIs. + <strong>Use cases</strong>: Ignite as an in-memory database or digital integration hub + with the active persistence layer. </p> </td> </tr> </tbody> </table> - <h2 style="padding-top: 10px;" id="collocated-processing"> - Collocated vs Client-Server Processing - <a href="/arch/memorycentric.html#collocated-processing"><i class="fa fa-anchor"></i></a> - </h2> - <p> - The disk-centric systems, like RDBMS or NoSQL, generally utilize the classic client-server approach, where - the data is brought from the server to the client side where it gets processed and then is usually discarded. - This approach does not scale well as moving the data over the network is the most expensive operation in a distributed system. - </p> - <p> - A much more scalable approach is <code>collocated processing</code> that reverses the flow by bringing the computations to the - servers where the data actually resides. This approach allows you to execute advanced logic or distributed SQL with JOINs - exactly where the data is stored avoiding expensive serialization and network trips. - </p> - <h2 style="padding-top: 10px;" id="partitioning"> Partitioning & Replication <a href="/arch/memorycentric.html#partitioning"><i class="fa fa-anchor"></i></a> </h2> <p> - Depending on the configuration, Ignite can either partition or replicate data across its memory-centric - storage. Unlike <code>REPLICATED</code> mode, where data is fully replicated across all nodes - in the cluster, in <code>PARTITIONED</code> mode Ignite will equally split the data across - multiple cluster nodes, allowing for storing TBs of data both in memory and on disk. + Depending on the configuration, Ignite can both partition or replicate data across its memory-centric + storage. With the replicated mode, each cluster node keeps a full copy of the data that is advantageous + for frequently accessed dictionaries. As for the partitioned mode, Ignite spreads the data across all + the cluster nodes evenly, allowing to store much more than a single machine can fit in. </p> <h2 style="padding-top: 10px;" id="durability"> - Durability + Durability and Consistency <a href="/arch/memorycentric.html#durability"><i class="fa fa-anchor"></i></a> </h2> <p> Ignite provides strong ACID durability guarantees to the data: + </p> <ul class="page-list" style="margin-bottom: 20px;"> - <li>Committed transactions will always survive any failures.</li> + <li>Committed transactions always survive failures.</li> <li> The cluster can always be recovered to the latest successfully committed transaction. </li> - <li> - The cluster restarts are very fast. - </li> </ul> - </p> - <h2 style="padding-top: 10px;" id="redundancy"> - Redundancy - <a href="/arch/memorycentric.html#redundancy"><i class="fa fa-anchor"></i></a> + + <h2 style="padding-top: 10px;" id="write-ahead-log"> + Write-Ahead Logging and Checkpointing + <a href="/arch/memorycentric.html#write-ahead-log"><i class="fa fa-anchor"></i></a> </h2> <p> - Ignite also allows to configure multiple <b>backup copies</b> to guarantee data resiliency - in case of failures. + If Ignite native persistence is selected as a disk tier, then every time a record is updated in memory, + the change is added to the write-ahead log (WAL). The purpose of the WAL is to propagate updates to + disk in the fastest way possible and provide a consistent recovery mechanism that supports full cluster + failures. </p> - <h2 style="padding-top: 10px;" id="consistency"> - Consistency - <a href="/arch/memorycentric.html#consistency"><i class="fa fa-anchor"></i></a> - </h2> <p> - Regardless of which replication scheme is used, Ignite guarantees data consistency across - all cluster members. + As WAL grows, it periodically gets checkpointed to the main storage. Checkpointing is the process of + copying dirty pages from the memory tier to the partition files on disk. A dirty page is a page that was + updated in memory, was appended to WAL, but was not written to a respective partition file on disk yet. </p> - <h2 style="padding-top: 10px;" id="write-ahead-log"> - Write-Ahead Log - <a href="/arch/memorycentric.html#write-ahead-log"><i class="fa fa-anchor"></i></a> - </h2> + <div class="page-heading">Learn More</div> <p> - Every time the data is updated in memory, the update will be appended to the tail of - the write-ahead log (WAL). The purpose of the WAL is to propagate updates to disk in - the fastest way possible and provide a consistent recovery mechanism that supports full cluster failures. + <a href="/arch/persistence.html"> + <b>Native Persistence <i class="fa fa-angle-double-right"></i></b> + </a> </p> - - <h2 style="padding-top: 10px;" id="checkpointing"> - Checkpointing - <a href="/arch/memorycentric.html#checkpointing"><i class="fa fa-anchor"></i></a> - </h2> <p> - As WAL grows, it periodically gets <i>checkpointed</i> to the main storage. - Checkpointing is the process of copying <i>dirty pages</i> from memory to the partition files on disk. - A dirty page is a page that was updated in memory, was appended to WAL, but was not written to a respective partition - file on disk yet. + <a href="/features/datagrid.html"> + <b>Ignite as an In-Memory Data Grid <i class="fa fa-angle-double-right"></i></b> + </a> </p> - - <h2 style="padding-top: 10px;" id="peristence-config"> - Persistence Configuration - <a href="/arch/memorycentric.html#peristence-config"><i class="fa fa-anchor"></i></a> - </h2> <p> - To enable Ignite persistence, add the following configuration parameter to the cluster's - node configuration: + <a href="/use-cases/in-memory-database.html"> + <b>Ignite as an In-Memory Database <i class="fa fa-angle-double-right"></i></b> + </a> + </p> + <p> + <a href="/use-cases/dih.html"> + <b>Ignite as a Digital Integration Hub <i class="fa fa-angle-double-right"></i></b> + </a> + </p> + <p> + <a href="/use-cases/hpc.html"> + <b>Ignite for High Performance Computing <i class="fa fa-angle-double-right"></i></b> + </a> </p> - <div class="tab-content"> - <div class="tab-pane active" id="configuration"> - <pre class="brush:xml"> - <bean class="org.apache.ignite.configuration.IgniteConfiguration"> - <!-- Enabling Apache Ignite native persistence. --> - <property name="dataStorageConfiguration"> - <bean class="org.apache.ignite.configuration.DataStorageConfiguration"> - <property name="defaultDataRegionConfiguration"> - <bean class="org.apache.ignite.configuration.DataRegionConfiguration"> - <property name="persistenceEnabled" value="true"/> - </bean> - </property> - </bean> - </property> - - <!-- Additional setting. --> - - </bean> - </pre> - </div> - </div><br/> - - <h2 style="padding-top: 10px;" id="more"> - More on Memory-Centric Storage - <a href="/arch/memorycentric.html#more"><i class="fa fa-anchor"></i></a> - </h2> - <table class="formatted" name="Deployment Options Features"> - <thead> - <tr> - <th width="35%" class="left">Feature</th> - <th>Description</th> - </tr> - </thead> - <tbody> - <tr> - <td class="left">Persistence</td> - <td> - <p> - Ignite native persistence is a distributed, ACID, and <nobr>SQL-compliant</nobr> disk store - that transparently integrates with Ignite memory-centric storage: - </p> - <div class="page-links"> - <a href="/arch/persistence.html">Docs for this feature <i class="fa fa-angle-double-right"></i></a> - </div> - </td> - </tr> - <tr> - <td class="left">Partitioning & Replication</td> - <td> - <p> - Depending on the configuration, Ignite can either <i>partition</i> or <i>replicate</i> - data. Unlike <code>REPLICATED</code> mode, where data is fully replicated across - all nodes in the cluster, in <code>PARTITIONED</code> mode Ignite will equally split the data - across multiple cluster nodes. - </p> - <div class="page-links"> - <a href="http://apacheignite.readme.io/docs/cache-modes" target="docs">Docs for this Feature <i class="fa fa-angle-double-right"></i></a> - </div> - </td> - </tr> - <tr> - <td class="left">Distributed Database</td> - <td> - <p> - Apache Ignite can be used as all-in-one distributed database that supports SQL, key-value, - compute, machine learning and other data processing APIs: - </p> - <div class="page-links"> - <a href="/use-cases/database/distributed-database.html">Docs for this feature <i class="fa fa-angle-double-right"></i></a> - </div> - </td> - </tr> - <tr> - <td class="left">In-Memory Database</td> - <td> - <p> - Apache Ignite can be used as a distributed and horizontally scalable in-memory database (IMDB): - </p> - <div class="page-links"> - <a href="/use-cases/database/in-memory-database.html">Docs for this feature <i class="fa fa-angle-double-right"></i></a> - </div> - </td> - </tr> - <tr> - <td class="left">Data Grid</td> - <td> - <p> - Ignite can act as a data grid that is a distributed, transactional key-value store. Unlike - other in-memory data grids (IMDG), Ignite enables storing data both, in memory and on disk, - and therefore is able to store more data than can fit in physical memory: - </p> - <div class="page-links"> - <a href="/features/datagrid.html">Docs for this Feature <i class="fa fa-angle-double-right"></i></a> - </div> - </td> - </tr> - <tr> - <td class="left">Database Caching</td> - <td> - <p> - Ignite is used as a caching layer (aka. data grid) above 3rd party databases such as RDBMS, - Apache Cassandra, MongoDB: - </p> - <div class="page-links"> - <a href="/use-cases/caching/database-caching.html">Docs for this Feature <i class="fa fa-angle-double-right"></i></a> - </div> - </td> - </tr> - </tbody> - </table> </section> </main> Modified: ignite/site/branches/ignite-redisign/includes/header.html URL: http://svn.apache.org/viewvc/ignite/site/branches/ignite-redisign/includes/header.html?rev=1873876&r1=1873875&r2=1873876&view=diff ============================================================================== --- ignite/site/branches/ignite-redisign/includes/header.html (original) +++ ignite/site/branches/ignite-redisign/includes/header.html Tue Feb 11 00:45:25 2020 @@ -36,19 +36,16 @@ <li class="dropdown" style="cursor: pointer;"> <a class="dropdown-toggle" data-toggle="dropdown" aria-label="Features">Features<span class="caret"></span></a> <ul class="dropdown-menu" role="menu"> - <li role="presentation" class="submenu-header" >Overview</li> - <li><a href="/whatisignite.html" aria-label="Overview" - onclick="ga('send', 'event', 'whatisignite', 'menu_click', 'whatisignite_page');"> - What is Apache Ignite®?</a> - </li> - <li class="divider"> - <!-- Ignite main features. --> <li role="presentation" class="submenu-header">Features</li> <li><a href="/arch/memorycentric.html" aria-label="Memory Centric Storage" onclick="ga('send', 'event', 'apache_ignite_features', 'menu_click', 'memory_centric');"> Memory-Centric Storage</a> </li> + <li><a href="/arch/persistence.html" + onclick="ga('send', 'event', 'apache_ignite_features', 'menu_click', 'persistence');"> + Native Persistence</a> + </li> <li><a href="/features/sql.html" aria-label="Features" onclick="ga('send', 'event', 'apache_ignite_features', 'menu_click', 'distributed_sql');"> Distributed SQL</a> @@ -57,10 +54,6 @@ onclick="ga('send', 'event', 'apache_ignite_features', 'menu_click', 'distributed_key_value');"> Distributed Key-Value</a> </li> - <li><a href="/arch/persistence.html" - onclick="ga('send', 'event', 'apache_ignite_features', 'menu_click', 'persistence');"> - Native Persistence</a> - </li> <li><a href="/features/transactions.html" aria-label="Transactions" onclick="ga('send', 'event', 'apache_ignite_features', 'menu_click', 'acid_transactions');"> ACID Transactions</a> @@ -73,10 +66,6 @@ onclick="ga('send', 'event', 'apache_ignite_features', 'menu_click', 'machine_learning');"> Machine Learning</a> </li> - <li><a href="/features/tensorflow.html" aria-label="Tensor Flow" - onclick="ga('send', 'event', 'apache_ignite_features', 'menu_click', 'tensorflow');"> - TensorFlow Integration</a> - </li> <li><a href="/features/multilanguage.html" aria-label="Multilanguage" onclick="ga('send', 'event', 'apache_ignite_features', 'menu_click', 'multi_language');"> Multi-Language</a> @@ -168,6 +157,10 @@ <li class="divider"> <li role="presentation" class="submenu-header">Learning Ignite</li> + <li><a href="/whatisignite.html" aria-label="Overview" + onclick="ga('send', 'event', 'whatisignite', 'menu_click', 'whatisignite_page');"> + FAQ</a> + </li> <li><a href="/blogs.html">Blogs</a></li> <li><a href="https://github.com/apache/ignite/tree/master/examples" target="_blank" rel="noopener"> Examples <i class="fa fa-external-link" style="padding-left:5px;"></i></a></li> Modified: ignite/site/branches/ignite-redisign/use-cases/hadoop-acceleration.html URL: http://svn.apache.org/viewvc/ignite/site/branches/ignite-redisign/use-cases/hadoop-acceleration.html?rev=1873876&r1=1873875&r2=1873876&view=diff ============================================================================== --- ignite/site/branches/ignite-redisign/use-cases/hadoop-acceleration.html (original) +++ ignite/site/branches/ignite-redisign/use-cases/hadoop-acceleration.html Tue Feb 11 00:45:25 2020 @@ -172,6 +172,11 @@ under the License. <b>Ignite DataFrames in Details <i class="fa fa-angle-double-right"></i></b> </a> </p> + <p> + <a href="/use-cases/dih.html"> + <b>Ignite as a Digital Integration Hub <i class="fa fa-angle-double-right"></i></b> + </a> + </p> </section> </main> Modified: ignite/site/branches/ignite-redisign/use-cases/in-memory-cache.html URL: http://svn.apache.org/viewvc/ignite/site/branches/ignite-redisign/use-cases/in-memory-cache.html?rev=1873876&r1=1873875&r2=1873876&view=diff ============================================================================== --- ignite/site/branches/ignite-redisign/use-cases/in-memory-cache.html (original) +++ ignite/site/branches/ignite-redisign/use-cases/in-memory-cache.html Tue Feb 11 00:45:25 2020 @@ -164,7 +164,7 @@ under the License. </a> </p> <p> - <a href="TODO"> + <a href="/use-cases/dih.html"> <b>Ignite as a Digital Integration Hub <i class="fa fa-angle-double-right"></i></b> </a> </p> Modified: ignite/site/branches/ignite-redisign/use-cases/in-memory-database.html URL: http://svn.apache.org/viewvc/ignite/site/branches/ignite-redisign/use-cases/in-memory-database.html?rev=1873876&r1=1873875&r2=1873876&view=diff ============================================================================== --- ignite/site/branches/ignite-redisign/use-cases/in-memory-database.html (original) +++ ignite/site/branches/ignite-redisign/use-cases/in-memory-database.html Tue Feb 11 00:45:25 2020 @@ -172,7 +172,7 @@ under the License. </a> </p> <p> - <a href="TODO"> + <a href="/use-cases/dih.html"> <b>Ignite as a Digital Integration Hub <i class="fa fa-angle-double-right"></i></b> </a> </p> Modified: ignite/site/branches/ignite-redisign/use-cases/spark-acceleration.html URL: http://svn.apache.org/viewvc/ignite/site/branches/ignite-redisign/use-cases/spark-acceleration.html?rev=1873876&r1=1873875&r2=1873876&view=diff ============================================================================== --- ignite/site/branches/ignite-redisign/use-cases/spark-acceleration.html (original) +++ ignite/site/branches/ignite-redisign/use-cases/spark-acceleration.html Tue Feb 11 00:45:25 2020 @@ -136,7 +136,11 @@ under the License. <b>Ignite DataFrames in Details <i class="fa fa-angle-double-right"></i></b> </a> </p> - + <p> + <a href="/use-cases/dih.html"> + <b>Ignite as a Digital Integration Hub <i class="fa fa-angle-double-right"></i></b> + </a> + </p> </section> </main>