pvary commented on code in PR #16961: URL: https://github.com/apache/iceberg/pull/16961#discussion_r4002896401
########## format/index-spec.md: ########## @@ -0,0 +1,734 @@ +--- +title: "Index Spec" +--- +<!-- + - Licensed to the Apache Software Foundation (ASF) under one or more + - contributor license agreements. See the NOTICE file distributed with + - this work for additional information regarding copyright ownership. + - The ASF licenses this file to You under the Apache License, Version 2.0 + - (the "License"); you may not use this file except in compliance with + - the License. You may obtain a copy of the License at + - + - http://www.apache.org/licenses/LICENSE-2.0 + - + - Unless required by applicable law or agreed to in writing, software + - distributed under the License is distributed on an "AS IS" BASIS, + - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + - See the License for the specific language governing permissions and + - limitations under the License. + --> +# Iceberg Index Specification + +## Background and Motivation + +An index is most valuable when it is a property of the table rather than of the engine that built it. This +specification defines a common format for index metadata and a common storage architecture for index data, so that any +engine can build an index, maintain it, and use it to plan queries against the table. + +## Goals + +* **Portability** -- An index written by one engine will be readable by any other engine. +* **Separation** -- Index metadata will be committed separately from table metadata. Building and maintaining an index + will not rewrite the table. +* **Optionality** -- Indexes will be optional. Engines may ignore an index they do not support. +* **Consistency** -- Each index snapshot will index exactly the live rows of one source table snapshot. + +## Overview + +An index is recorded in an index metadata file that contains the index definition and a set of index snapshots. Each +index snapshot corresponds to a snapshot of the source table and references the index data for that state. + +Index metadata files and index data files are immutable. Every update writes a new metadata file and a new tracking Review Comment: Done ########## format/index-spec.md: ########## @@ -0,0 +1,734 @@ +--- +title: "Index Spec" +--- +<!-- + - Licensed to the Apache Software Foundation (ASF) under one or more + - contributor license agreements. See the NOTICE file distributed with + - this work for additional information regarding copyright ownership. + - The ASF licenses this file to You under the Apache License, Version 2.0 + - (the "License"); you may not use this file except in compliance with + - the License. You may obtain a copy of the License at + - + - http://www.apache.org/licenses/LICENSE-2.0 + - + - Unless required by applicable law or agreed to in writing, software + - distributed under the License is distributed on an "AS IS" BASIS, + - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + - See the License for the specific language governing permissions and + - limitations under the License. + --> +# Iceberg Index Specification + +## Background and Motivation + +An index is most valuable when it is a property of the table rather than of the engine that built it. This +specification defines a common format for index metadata and a common storage architecture for index data, so that any +engine can build an index, maintain it, and use it to plan queries against the table. + +## Goals + +* **Portability** -- An index written by one engine will be readable by any other engine. +* **Separation** -- Index metadata will be committed separately from table metadata. Building and maintaining an index + will not rewrite the table. +* **Optionality** -- Indexes will be optional. Engines may ignore an index they do not support. +* **Consistency** -- Each index snapshot will index exactly the live rows of one source table snapshot. + +## Overview + +An index is recorded in an index metadata file that contains the index definition and a set of index snapshots. Each +index snapshot corresponds to a snapshot of the source table and references the index data for that state. + +Index metadata files and index data files are immutable. Every update writes a new metadata file and a new tracking +file, may reuse existing range files, and is committed by an atomic swap of the index metadata file, as defined in +[Commits and Concurrency](#commits-and-concurrency). + +The index data of a snapshot is organized as a [tracking file](#tracking-file) that lists a set of +[range files](#range-files): + +```text +Index Metadata + | + +-- Index Snapshot(s) + | + +-- Tracking File + | + +-- Range Files +``` + +## Specification + +### Terms + +* **Index** -- A structure that accelerates retrieval of rows from a source table. +* **Index snapshot** -- The state of an index for a single snapshot of the source table. +* **Index entry** -- The values produced by the index fields for one indexed row of the source table. +* **Clustering key** -- The tuple of values that determines the position of an index entry within an index snapshot. +* **Tracking file** -- A file that lists the range files of an index snapshot; one per index snapshot. +* **Range file** -- A file that stores the index entries for a range of clustering keys; a subset of an index snapshot. Review Comment: My mistake. Fixed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
