This is an automated email from the ASF dual-hosted git repository.
JingsongLi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/paimon.git
The following commit(s) were added to refs/heads/master by this push:
new c78e9626ab docs: clarify Data Evolution storage semantics
c78e9626ab is described below
commit c78e9626abed98273bb3fd0c1293ec18439295e8
Author: JingsongLi <[email protected]>
AuthorDate: Tue Oct 27 13:29:14 2026 +0800
docs: clarify Data Evolution storage semantics
---
AGENTS.md | 27 +++++++++++++++++++++++++++
README.md | 9 ++++++---
docs/docs/concepts/basic-concepts.md | 19 +++++++++++++++++--
docs/docs/concepts/spec/datafile.md | 13 +++++++++++++
4 files changed, 63 insertions(+), 5 deletions(-)
diff --git a/AGENTS.md b/AGENTS.md
index 2e4ecff82e..8d85850442 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -2,6 +2,33 @@
Guidelines for AI coding assistants working with the Apache Paimon codebase.
+## Architecture and Table Modes
+
+Identify the table mode before explaining storage, reads, updates, or
compaction. Paimon shares
+schema, snapshot, and manifest metadata across different data layouts; it must
not be described
+as universally LSM-based.
+
+| Mode | Data organization and read semantics |
+| --- | --- |
+| Ordinary append table | Stores inserted rows without primary-key merging. |
+| Primary-key table | Uses an LSM-based layout to reconcile record versions by
key. |
+| Data Evolution append table | Aligns column files by row-ID range and
selects current column values by file version, independently of primary-key LSM
merging. |
+
+Data Evolution requires `data-evolution.enabled=true`,
`row-tracking.enabled=true`, no primary
+key, and `bucket=-1`. It supports partial-column updates that retain files for
untouched columns.
+Its column merging and compaction must not be explained as primary-key LSM
merging. A
+`bucket-0` directory, sequence-number metadata, or compaction alone does not
imply an LSM layout.
+
+For architecture and file-layout comparisons, cover Data Evolution explicitly
rather than
+using the primary-key layout as the model for all Paimon tables.
+Distinguish table metadata and row reconstruction from the configurable
physical file format.
+
+Start with [Basic Concepts](docs/docs/concepts/basic-concepts.md),
+[Data Evolution](docs/docs/multimodal-table/data-evolution.mdx), and
+[File Layout and
Reads](docs/docs/multimodal-table/data-evolution-file-layout.mdx).
+Verify implementation details in `AppendOnlyFileStoreTable`,
`DataEvolutionSplitRead`,
+`DataEvolutionReadPlanner`, and `SchemaValidation` under
`paimon-core/src/main/java/org/apache/paimon/`.
+
## Syntax Requirements
- Based on JDK 8 and Scala 2.12, higher version syntax features must not be
used.
diff --git a/README.md b/README.md
index e257fc48ff..eb34b5de0d 100644
--- a/README.md
+++ b/README.md
@@ -3,9 +3,12 @@
[](https://www.apache.org/licenses/LICENSE-2.0.html)
[](https://the-asf.slack.com/archives/C053Q2NCW8G)
-Apache Paimon is a lake format that enables building a Realtime Lakehouse
Architecture with Flink and Spark
-for both streaming and batch operations. Paimon innovatively combines lake
format and LSM structure, bringing realtime
-streaming updates into the lake architecture.
+Apache Paimon is a lake format for streaming and batch workloads with Flink
and Spark.
+Append and primary-key tables share versioned snapshots and manifest-based
file management.
+Primary-key tables use an LSM-based layout for streaming updates. Append
tables can enable
+[Data Evolution](docs/docs/multimodal-table/data-evolution.mdx) to align
column files by row ID
+and update selected columns while retaining files for untouched columns. Data
Evolution uses
+positional column merging independently of primary-key LSM merging.
Background and documentation are available at https://paimon.apache.org
diff --git a/docs/docs/concepts/basic-concepts.md
b/docs/docs/concepts/basic-concepts.md
index 8397badd3a..54b91a2a17 100644
--- a/docs/docs/concepts/basic-concepts.md
+++ b/docs/docs/concepts/basic-concepts.md
@@ -29,15 +29,22 @@ snapshots. This page introduces the logical table first,
then explains how it ma
## Table Types
-| Table type | Logical behavior | Typical use |
+| Table type or storage mode | Logical behavior | Typical use |
| --- | --- | --- |
| [Append table](../append-table/) | Stores incoming rows without merging them
by primary key. | Event logs and batch datasets |
| [Primary-key table](../primary-key-table/) | Merges records with the same
key according to the configured merge engine. | CDC, upserts, and updates to an
entity |
+| [Data Evolution append table](../multimodal-table/data-evolution) | Aligns
column files by row ID and selects the current version of each column without
primary-key LSM merging. | Column backfills, feature engineering, and
multimodal enrichment |
A primary-key table can keep multiple physical versions of a record. Reads and
compaction
reconcile those versions to produce the logical result. An append table does
not merge incoming
records by key; supported engines can still perform [batch row-level
operations](../append-table/#row-level-operations).
+Data Evolution is an append-table storage mode, not a primary-key merge
engine. It requires
+`data-evolution.enabled = true`, `row-tracking.enabled = true`, no primary
key, and `bucket = -1`.
+It can update selected columns while retaining files for untouched columns.
Readers reconstruct
+rows from row-ID-aligned column files; see [File Layout and
Reads](../multimodal-table/data-evolution-file-layout).
+Paimon's shared snapshot and manifest structure does not imply that all tables
use an LSM layout.
+
[Multimodal tables](../multimodal-table/) add storage capabilities for data
such as BLOBs and
vectors. The [REST Catalog table guide](./rest/tables) also describes Format
Tables and Object
Tables, which expose other kinds of stored data through the catalog.
@@ -66,6 +73,9 @@ A bucket groups files within a partition. Its role depends on
the table's bucket
- **Dynamic buckets** assign keys to buckets and track the assignment in an
index for primary-key tables.
- **Bucket-unaware append tables** use a `bucket-0` directory without tying
writer parallelism to a fixed bucket count.
+Data Evolution uses the bucket-unaware append mode. Its `bucket-0` directory
does not contain
+a primary-key LSM tree; column files are associated through row-ID ranges and
file versions.
+
See [primary-key data distribution](../primary-key-table/data-distribution)
for the complete
set of modes and [bucketed append tables](../append-table/bucketed) for
append-table choices.
@@ -116,8 +126,13 @@ Data files contain the records. Parquet is the default
format; see [File Format]
for the available formats and type mappings. Primary-key data files also carry
information used
to merge records, such as row kind and sequence number.
+Data Evolution files can supply different columns for the same row-ID range. A
reader selects
+the current providers for the requested columns and combines their aligned row
positions.
+File sequence numbers identify column versions here, not versions of a
business primary key.
+
Compaction replaces a set of data files with a new set. It can consolidate
small files and,
-for primary-key tables, merge record versions. Changelog files serve
incremental readers when
+for primary-key tables, merge record versions. Data Evolution compaction
consolidates column
+files while preserving row reconstruction. Changelog files serve incremental
readers when
the configured [changelog producer](../primary-key-table/changelog-producer)
generates them.
## Follow a Record Through Snapshots
diff --git a/docs/docs/concepts/spec/datafile.md
b/docs/docs/concepts/spec/datafile.md
index 240335200d..25408c53d1 100644
--- a/docs/docs/concepts/spec/datafile.md
+++ b/docs/docs/concepts/spec/datafile.md
@@ -90,6 +90,19 @@ depends on the table mode and enabled features. The examples
below use ordinary
[row tracking](../../append-table/row-tracking) and
[data evolution](../../multimodal-table/data-evolution) can add metadata or
split columns across files.
+### Data Evolution Append Table Files
+
+Data Evolution is an append-table storage mode, independent of primary-key LSM
merging. It
+requires row tracking, no primary key, and `bucket = -1`. Its `bucket-0`
directory can contain
+multiple files supplying different columns for the same logical rows.
+
+Normal files have equal or disjoint row-ID ranges. Readers select the newest
normal file
+providing each requested column and combine columns at matching row positions.
Dedicated
+BLOB and vector files must fit within one normal range. The metadata includes
`firstRowId`,
+`rowCount`, `schemaId`, written columns, and file sequence numbers. See
+[File Layout and Reads](../../multimodal-table/data-evolution-file-layout) for
the complete
+range and column-version contracts, including dedicated-file coverage and BLOB
fallback rules.
+
### Table with Primary key Data File
For the standard primary-key layout, fields appear in this order:
