Nikita-tech-writer commented on a change in pull request #9708: URL: https://github.com/apache/ignite/pull/9708#discussion_r781211967
########## File path: docs/_docs/persistence/change-data-capture.adoc ########## @@ -0,0 +1,128 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to You under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. += Change Data Capture + +WARNING: Experimental API. + +== Overview +Change Data Capture (link:https://en.wikipedia.org/wiki/Change_data_capture[CDC]) is data processing pattern used to asynchronously receive entries that has changed on local node +so that action can be taken using the changed entry. + +Some use-cases for CDC: + + * Stream changes in Warehouse. + * Update search index. + * Calculate statistics (streaming queries). + * Audit logs. + * Async interaction with extenal system - moderation, business process invocation, etc. + +Ignite implements CDC with an application `ignite-cdc.sh` and link:https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/cdc/CdcConsumer.java#L56[Java API]. + +CDC application and Ignite node integrated via WAL archive segments: + +image:../../assets/images/integrations/CDC-design.svg[] + +When CDC enabled Ignite server node creates a hard link to each WAL archive segment in special `db/cdc/\{consistency_id\}` directory. +`ignite-cdc.sh` run in different JVM and process newly archived link:native-persistence.adoc#_write-ahead_log[WAL segments]. +When segment fully processed by `ignite-cdc.sh` it removed. The actual disk space free when both links - archive and CDC removed. +tate of consumption can be saved to disk to continue from it in the case of any failure. + +== Configuration + +=== Ignite node: + +[cols="20%,45%,35%",opts="header"] +|=== +|Name |Description | Default value +| `DataStorageConfiguration#cdcEnabled` | Flag to enable CDC on server node. | `false` +| `DataStorageConfiguration#cdcWalPath` | Path to the CDC directory | `"db/wal/cdc"` +| `DataStorageConfiguration#walForceArchiveTimeout` | Timeout to forcefully archive WAL segment even it not full. | `-1` (disabled) +|=== + +=== CDC application: + +CDC configured in the same manner as Ignite node via spring XML file. +`ignite-cdc.sh` require both Ignite and CDC configuration to start. +`IgniteConfiguration` used to determine common options like path to CDC directory, node consistent id and others. +And `CdcConfiguration` contains options specifically for `ignite-cdc.sh`. Review comment: ```suggestion Change Data Capture is configured in the same way as the Ignite node - via the spring XML file: * `ignite-cdc.sh` requires both Ignite and Change Data Capture configurations to start; * `IgniteConfiguration` is used to determine common options like a path to the Change Data Capture directory, node consistent id, and other parameters; * `CdcConfiguration` contains `ignite-cdc.sh`-specific options. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
