This is an automated email from the ASF dual-hosted git repository.
blue pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/iceberg-docs.git
The following commit(s) were added to refs/heads/main by this push:
new 35aa365 Updating common pages (#89)
35aa365 is described below
commit 35aa3656662d3573e892c8e09fc235478ef10c9b
Author: Samuel Redai <[email protected]>
AuthorDate: Fri Jun 3 08:31:20 2022 -0700
Updating common pages (#89)
---
landing-page/content/common/spec.md | 90 ++++++++--
landing-page/content/common/view-spec.md | 272 +++++++++++++++++++++++++++++++
2 files changed, 347 insertions(+), 15 deletions(-)
diff --git a/landing-page/content/common/spec.md
b/landing-page/content/common/spec.md
index 8fc0612..a2676e2 100644
--- a/landing-page/content/common/spec.md
+++ b/landing-page/content/common/spec.md
@@ -54,7 +54,7 @@ In addition to row-level deletes, version 2 makes some
requirements stricter for
* **Evolution** -- Tables will support full schema and partition spec
evolution. Schema evolution supports safe column add, drop, reorder and rename,
including in nested structures.
* **Dependable types** -- Tables will provide well-defined and dependable
support for a core set of types.
* **Storage separation** -- Partitioning will be table configuration. Reads
will be planned using predicates on data values, not partition values. Tables
will support evolving partition schemes.
-* **Formats** -- Underlying data file formats will support identical schema
evolution rules and types. Both read- and write-optimized formats will be
available.
+* **Formats** -- Underlying data file formats will support identical schema
evolution rules and types. Both read-optimized and write-optimized formats will
be available.
## Overview
@@ -159,7 +159,7 @@ For the representations of these types in Avro, ORC, and
Parquet file formats, s
#### Nested Types
-A **`struct`** is a tuple of typed values. Each field in the tuple is named
and has an integer id that is unique in the table schema. Each field can be
either optional or required, meaning that values can (or cannot) be null.
Fields may be any type. Fields may have an optional comment or doc string.
+A **`struct`** is a tuple of typed values. Each field in the tuple is named
and has an integer id that is unique in the table schema. Each field can be
either optional or required, meaning that values can (or cannot) be null.
Fields may be any type. Fields may have an optional comment or doc string.
Fields can have [default values](#default-values).
A **`list`** is a collection of values with some element type. The element
field has an integer id that is unique in the table schema. Elements can be
either optional or required. Element types may be any type.
@@ -195,6 +195,19 @@ Notes:
For details on how to serialize a schema to JSON, see Appendix C.
+#### Default values
+
+Default values can be tracked for struct fields (both nested structs and the
top-level schema's struct). There can be two defaults with a field:
+- `initial-default` is used to populate the field's value for all records that
were written before the field was added to the schema
+- `write-default` is used to populate the field's value for any records
written after the field was added to the schema, if the writer does not supply
the field's value
+
+The `initial-default` is set only when a field is added to an existing schema.
The `write-default` is initially set to the same value as `initial-default` and
can be changed through schema evolution. If either default is not set for an
optional field, then the default value is null for compatibility with older
spec versions.
+
+The `initial-default` and `write-default` produce SQL default value behavior,
without rewriting data files. SQL default value behavior when a field is added
handles all existing rows as though the rows were written with the new field's
default value. Default value changes may only affect future records and all
known fields are written into data files. Omitting a known field when writing a
data file is never allowed. The write default for a field must be written if a
field is not supplied [...]
+
+Default values are attributes of fields in schemas and serialized with fields
in the JSON format. See [Appendix C](#appendix-c-json-serialization).
+
+
#### Schema Evolution
Schemas may be evolved by type promotion or adding, deleting, renaming, or
reordering fields in structs (both nested structs and the top-level schema’s
struct).
@@ -211,6 +224,15 @@ Any struct, including a top-level schema, can evolve
through deleting fields, ad
Grouping a subset of a struct’s fields into a nested struct is **not**
allowed, nor is moving fields from a nested struct into its immediate parent
struct (`struct<a, b, c> ↔ struct<a, struct<b, c>>`). Evolving primitive types
to structs is **not** allowed, nor is evolving a single-field struct to a
primitive (`map<string, int> ↔ map<string, struct<int>>`).
+Struct evolution requires the following rules for default values:
+* The `initial-default` must be set when a field is added and cannot change
+* The `write-default` must be set when a field is added and may change
+* When a required field is added, both defaults must be set to a non-null value
+* When an optional field is added, the defaults may be null and should be
explicitly set
+* When a new field is added to a struct with a default value, updating the
struct's default is optional
+* If a field value is missing from a struct's `initial-default`, the field's
`initial-default` must be used for the field
+* If a field value is missing from a struct's `write-default`, the field's
`write-default` must be used for the field
+
#### Column Projection
@@ -390,7 +412,7 @@ The schema of a manifest file is a struct called
`manifest_entry` with the follo
| v1 | v2 | Field id, name | Type
| Description
|
| ---------- | ----------
|--------------------------|-----------------------------------------------------------|---------------------------------------------------------------------------------------|
-| _required_ | _required_ | **`0 status`** | `int` with meaning: `0:
EXISTING` `1: ADDED` `2: DELETED` | Used to track additions and deletions
|
+| _required_ | _required_ | **`0 status`** | `int` with meaning: `0:
EXISTING` `1: ADDED` `2: DELETED` | Used to track additions and deletions.
Deletes are informational only and not used in scans.
|
| _required_ | _optional_ | **`1 snapshot_id`** | `long`
| Snapshot id where the file was added, or
deleted if status is 2. Inherited when null. |
| | _optional_ | **`3 sequence_number`** | `long`
| Sequence number when the file was added.
Inherited when null. |
| _required_ | _required_ | **`2 data_file`** | `data_file` `struct`
(see below) | File path, partition tuple, metrics, ...
|
@@ -423,8 +445,9 @@ The schema of a manifest file is a struct called
`manifest_entry` with the follo
Notes:
1. Single-value serialization for lower and upper bounds is detailed in
Appendix D.
-2. For `float` and `double`, the value `-0.0` must precede `+0.0`, as in the
IEEE 754 `totalOrder` predicate.
+2. For `float` and `double`, the value `-0.0` must precede `+0.0`, as in the
IEEE 754 `totalOrder` predicate. NaNs are not permitted as lower or upper
bounds.
3. If sort order ID is missing or unknown, then the order is assumed to be
unsorted. Only data files and equality delete files should be written with a
non-null order id. [Position deletes](#position-delete-files) are required to
be sorted by file and position, not a table order, and should set sort order id
to null. Readers must ignore sort order id for position delete files.
+4. The following field ids are reserved on `data_file`: 141.
The `partition` struct stores the tuple of partition values for each file. Its
type is derived from the partition fields of the partition spec used to write
the manifest file. In v2, the partition struct's field ids must match the ids
from the partition spec.
@@ -443,7 +466,7 @@ Iceberg v2 adds a sequence number to the entry and makes
the snapshot id optiona
Notes:
-1. Technically, data files can be deleted when the last snapshot that contains
the file as “live” data is garbage collected. But this is harder to detect and
requires finding the diff of multiple snapshots. It is easier to track what
files are deleted in a snapshot and delete them when that snapshot expires.
+1. Technically, data files can be deleted when the last snapshot that contains
the file as “live” data is garbage collected. But this is harder to detect and
requires finding the diff of multiple snapshots. It is easier to track what
files are deleted in a snapshot and delete them when that snapshot expires. It
is not recommended to add a deleted file back to a table. Adding a deleted file
can lead to edge cases where incremental deletes can break table snapshots.
2. Manifest list files are required in v2, so that the `sequence_number` and
`snapshot_id` to inherit are always available.
#### Sequence Number Inheritance
@@ -507,7 +530,7 @@ Manifest list files store `manifest_file`, a struct with
the following fields:
| v1 | v2 | Field id, name | Type
| Description |
| ---------- | ----------
|--------------------------------|---------------------------------------------|-------------|
| _required_ | _required_ | **`500 manifest_path`** | `string`
| Location of the manifest file |
-| _required_ | _required_ | **`501 manifest_length`** | `long`
| Length of the manifest file |
+| _required_ | _required_ | **`501 manifest_length`** | `long`
| Length of the manifest file in bytes |
| _required_ | _required_ | **`502 partition_spec_id`** | `int`
| ID of a partition spec used to write the manifest;
must be listed in table metadata `partition-specs` |
| | _required_ | **`517 content`** | `int` with
meaning: `0: data`, `1: deletes` | The type of files tracked by the manifest,
either data or delete files; 0 for all v1 manifests |
| | _required_ | **`515 sequence_number`** | `long`
| The sequence number when the manifest was added to
the table; use 0 when reading v1 manifest lists |
@@ -538,7 +561,7 @@ Notes:
#### Scan Planning
-Scans are planned by reading the manifest files for the current snapshot.
Deleted entries in data and delete manifests are not used in a scan.
+Scans are planned by reading the manifest files for the current snapshot.
Deleted entries in data and delete manifests (those marked with status
"DELETED") are not used in a scan.
Manifests that contain no matching files, determined using either file counts
or partition summaries, may be skipped.
@@ -550,7 +573,10 @@ For example, an `events` table with a timestamp column
named `ts` that is partit
Scan predicates are also used to filter data and delete files using column
bounds and counts that are stored by field id in manifests. The same filter
logic can be used for both data and delete files because both store metrics of
the rows either inserted or deleted. If metrics show that a delete file has no
rows that match a scan predicate, it may be ignored just as a data file would
be ignored [2].
-Data files that match the query filter must be read by the scan.
+Data files that match the query filter must be read by the scan.
+
+Note that for any snapshot, all file paths marked with "ADDED" or "EXISTING"
may appear at most once across all manifest files in the snapshot. If a file
path appears more then once, the results of the scan are undefined. Reader
implementations may raise an error in this case, but are not required to do so.
+
Delete files that match the query filter must be applied to data files at read
time, limited by the scope of the delete file using the following rules.
@@ -673,11 +699,11 @@ Notes:
The atomic swap needed to commit new versions of table metadata can be
implemented by storing a pointer in a metastore or database that is updated
with a check-and-put operation [1]. The check-and-put validates that the
version of the table that a write is based on is still current and then makes
the new metadata from the write the current version.
-Each version of table metadata is stored in a metadata folder under the
table’s base location using a naming scheme that includes a version and UUID:
`<V>-<uuid>.metadata.json`. To commit a new metadata version, `V+1`, the writer
performs the following steps:
+Each version of table metadata is stored in a metadata folder under the
table’s base location using a naming scheme that includes a version and UUID:
`<V>-<random-uuid>.metadata.json`. To commit a new metadata version, `V+1`, the
writer performs the following steps:
-2. Create a new table metadata file based on the current metadata.
-3. Write the new table metadata to a unique file: `<V+1>-<uuid>.metadata.json`.
-4. Request that the metastore swap the table’s metadata pointer from the
location of `V` to the location of `V+1`.
+1. Create a new table metadata file based on the current metadata.
+2. Write the new table metadata to a unique file:
`<V+1>-<random-uuid>.metadata.json`.
+3. Request that the metastore swap the table’s metadata pointer from the
location of `V` to the location of `V+1`.
1. If the swap succeeds, the commit succeeded. `V` was still the latest
metadata version and the metadata file for `V+1` is now the current metadata.
2. If the swap fails, another writer has already created `V+1`. The
current writer goes back to step 1.
@@ -838,7 +864,7 @@ Note that the string map case is for maps where the key
type is a string. Using
Values should be stored in Parquet using the types and logical type
annotations in the table below. Column IDs are required.
-Lists must use the [3-level
representation](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#lists).
+Lists must use the [3-level
representation](https://github.com/apache/parquet-format/blob/master/LogicalTypes#lists).
| Type | Parquet physical type
| Logical type | Notes
|
|--------------------|--------------------------------------------------------------------|---------------------------------------------|----------------------------------------------------------------|
@@ -966,10 +992,12 @@ Types are serialized according to this table:
|**`fixed(L)`**|`JSON string: "fixed[<L>]"`|`"fixed[16]"`|
|**`binary`**|`JSON string: "binary"`|`"binary"`|
|**`decimal(P, S)`**|`JSON string: "decimal(<P>,<S>)"`|`"decimal(9,2)"`,<br
/>`"decimal(9, 2)"`|
-|**`struct`**|`JSON object: {`<br /> `"type": "struct",`<br
/> `"fields": [ {`<br /> `"id": <field id
int>,`<br /> `"name": <name string>,`<br
/> `"required": <boolean>,`<br
/> `"type": <type JSON>,`<br
/> `"doc": <comment string>`<br
/> `}, ...`<br /> `] }`|`{`<br
/> `"type": "struct",`<br /> `"fi [...]
+|**`struct`**|`JSON object: {`<br /> `"type": "struct",`<br
/> `"fields": [ {`<br /> `"id": <field id
int>,`<br /> `"name": <name string>,`<br
/> `"required": <boolean>,`<br
/> `"type": <type JSON>,`<br
/> `"doc": <comment string>,`<br
/> `"initial-default": <JSON encoding of default
value>,`<br /> `"write-d [...]
|**`list`**|`JSON object: {`<br /> `"type": "list",`<br
/> `"element-id": <id int>,`<br /> `"element-required":
<bool>`<br /> `"element": <type JSON>`<br />`}`|`{`<br
/> `"type": "list",`<br /> `"element-id": 3,`<br
/> `"element-required": true,`<br /> `"element":
"string"`<br />`}`|
|**`map`**|`JSON object: {`<br /> `"type": "map",`<br
/> `"key-id": <key id int>,`<br /> `"key": <type
JSON>,`<br /> `"value-id": <val id int>,`<br
/> `"value-required": <bool>`<br /> `"value": <type
JSON>`<br />`}`|`{`<br /> `"type": "map",`<br
/> `"key-id": 4,`<br /> `"key": "string",`<br
/> `"value-id": 5,`<br /> `"value-required": false,`<br
/> `"value": [...]
+Note that default values are serialized using the JSON single-value
serialization in [Appendix D](#appendix-d-single-value-serialization).
+
### Partition Specs
@@ -985,7 +1013,7 @@ Each partition field in the fields list is stored as an
object. See the table fo
|Transform or Field|JSON representation|Example|
|--- |--- |--- |
|**`identity`**|`JSON string: "identity"`|`"identity"`|
-|**`bucket[N]`**|`JSON string: "bucket<N>]"`|`"bucket[16]"`|
+|**`bucket[N]`**|`JSON string: "bucket[<N>]"`|`"bucket[16]"`|
|**`truncate[W]`**|`JSON string: "truncate[<W>]"`|`"truncate[20]"`|
|**`year`**|`JSON string: "year"`|`"year"`|
|**`month`**|`JSON string: "month"`|`"month"`|
@@ -1070,6 +1098,8 @@ Example
## Appendix D: Single-value serialization
+### Binary single-value serialization
+
This serialization scheme is for storing single values as individual binary
values in the lower and upper bounds maps of manifest files.
| Type | Binary serialization
|
@@ -1092,9 +1122,39 @@ This serialization scheme is for storing single values
as individual binary valu
| **`list`** | Not supported
|
| **`map`** | Not supported
|
+### JSON single-value serialization
+
+ Single values are serialized as JSON by type according to the following table:
+
+| Type | JSON representation | Example
| Description
|
+| ------------------ | ----------------------------------------- |
------------------------------------------ | -- |
+| **`boolean`** | **`JSON boolean`** | `true`
| |
+| **`int`** | **`JSON int`** | `34`
| |
+| **`long`** | **`JSON long`** | `34`
| |
+| **`float`** | **`JSON number`** | `1.0`
| |
+| **`double`** | **`JSON number`** | `1.0`
| |
+| **`decimal(P,S)`** | **`JSON number`** | `14.20`
| Stores the decimal as a number with S places
after the decimal |
+| **`date`** | **`JSON string`** |
`"2017-11-16"` | Stores ISO-8601 standard date |
+| **`time`** | **`JSON string`** |
`"22:31:08.123456"` | Stores ISO-8601 standard time with
microsecond precision |
+| **`timestamp`** | **`JSON string`** |
`"2017-11-16T22:31:08.123456"` | Stores ISO-8601 standard timestamp
with microsecond precision; must not include a zone offset |
+| **`timestamptz`** | **`JSON string`** |
`"2017-11-16T22:31:08.123456-07:00"` | Stores ISO-8601 standard timestamp
with microsecond precision; must include a zone offset |
+| **`string`** | **`JSON string`** | `"iceberg"`
| |
+| **`uuid`** | **`JSON string`** |
`"f79c3e09-677c-4bbd-a479-3f349cb785e7"` | Stores the lowercase uuid string |
+| **`fixed(L)`** | **`JSON string`** |
`"0x00010203"` | Stored as a hexadecimal string,
prefixed by `0x` |
+| **`binary`** | **`JSON string`** |
`"0x00010203"` | Stored as a hexadecimal string,
prefixed by `0x` |
+| **`struct`** | **`JSON object by field ID`** | `{"1": 1,
"2": "bar"}` | Stores struct fields using the field ID as
the JSON field name; field values are stored using this JSON single-value
format |
+| **`list`** | **`JSON array of values`** | `[1, 2, 3]`
| Stores a JSON array of values that are
serialized using this JSON single-value format |
+| **`map`** | **`JSON object of key and value arrays`** | `{ "keys":
["a", "b"], "values": [1, 2] }` | Stores arrays of keys and values; individual
keys and values are serialized using this JSON single-value format |
+
## Appendix E: Format version changes
+### Version 3
+
+Default values are added to struct fields in v3.
+* The `write-default` is a forward-compatible change because it is only used
at write time. Old writers will fail because the field is missing.
+* Tables with `initial-default` will be read correctly by older readers if
`initial-default` is always null for optional fields. Otherwise, old readers
will default optional columns with null. Old readers will fail to read required
fields which are populated by `initial-default` because that default is not
supported.
+
### Version 2
Writing v1 metadata:
diff --git a/landing-page/content/common/view-spec.md
b/landing-page/content/common/view-spec.md
new file mode 100644
index 0000000..eb89f7a
--- /dev/null
+++ b/landing-page/content/common/view-spec.md
@@ -0,0 +1,272 @@
+---
+url: view-spec
+toc: true
+aliases:
+ - "view-spec"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Iceberg View Spec
+
+## Background and Motivation
+
+Most compute engines (e.g. Trino and Apache Spark) support views. A view is a
logical table that can be referenced by future queries. Views do not contain
any data. Instead, the query stored by the view is executed every time the view
is referenced by another query.
+
+Each compute engine stores the metadata of the view in its proprietary format
in the metastore of choice. Thus, views created from one engine can not be read
or altered easily from another engine even when engines share the metastore as
well as the storage system. This document standardizes the view metadata for
ease of sharing the views across engines.
+
+## Goals
+
+* A common metadata format for view metadata, similar to how Iceberg supports
a common table format for tables.
+* The view metadata format specification
+ * Includes storage format as well as APIs to write/read the metadata.
+ * Supports versioning of views to track how a view evolved over time.
+
+## Overview
+
+View metadata storage mirrors how Iceberg table metadata is stored and
retrieved. View metadata is maintained in metadata files. All changes to view
state create a new view metadata file and completely replace the old metadata
using an atomic swap. Like Iceberg tables, this atomic swap is delegated to the
metastore that tracks tables and/or views by name. The view metadata file
tracks the view schema, custom properties, current and past versions, as well
as other metadata.
+Each metadata file is self-sufficient. It contains the history of the last few
operations performed on the view and can be used to roll back the view to a
previous version.
+
+### Metadata Location
+
+An atomic swap of one view metadata file for another provides the basis for
making atomic changes. Readers use the version of the view that was current
when they loaded the view metadata and are not affected by changes until they
refresh and pick up a new metadata location.
+
+Writers create view metadata files optimistically, assuming that the current
metadata location will not be changed before the writer’s commit. Once a writer
has created an update, it commits by swapping the view's metadata file pointer
from the base location to the new location.
+
+## Specification
+
+### Terms
+
+* **Schema** -- Names and types of fields in a view.
+* **Version** -- The state of a view at some point in time.
+
+### View Metadata
+
+The view version metadata file has the following fields:
+
+| Required/Optional | Field Name | Description |
+|-------------------|------------|-------------|
+| Required | format-version | An integer version number for the view format.
Currently, this must be 1. Implementations must throw an exception if the
view's version is higher than the supported version. |
+| Required | location | The view's base location. This is used to determine
where to store manifest files and view metadata files. |
+| Required | current-version-id | Current version of the view. Set to ‘1’ when
the view is first created. |
+| Optional | properties | A string to string map of view properties. This is
used for metadata such as "comment" and for settings that affect view
maintenance. This is not intended to be used for arbitrary metadata. |
+| Required | versions | An array of structs describing the last known versions
of the view. Controlled by the table property: “version.history.num-entries”.
See section [Versions](#versions). |
+| Required | version-log | A list of timestamp and version ID pairs that
encodes changes to the current version for the view. Each time the
current-version-id is changed, a new entry should be added with the
last-updated-ms and the new current-version-id. |
+| Optional | schemas | A list of schemas, the same as the ‘schemas’ field from
Iceberg table spec. |
+| Optional | current-schema-id | ID of the current schema of the view |
+
+#### Versions
+
+Field "versions" is an array of structs with the following fields:
+
+| Required/Optional | Field Name | Description |
+|-------------------|------------|-------------|
+| Required | version-id | Monotonically increasing id indicating the version
of the view. Starts with 1. |
+| Required | timestamp-ms | Timestamp expressed in ms since epoch at which the
version of the view was created. |
+| Required | summary | A string map summarizes the version changes, including
`operation`, described in [Summary](#summary). |
+| Required | representations | A list of "representations" as described in
[Representations](#representations). |
+
+#### Version Log
+
+Field “version-log” is an array of structs that describe when each version was
considered "current". Creation time is different and is stored in each
version's metadata. This allows you to reconstruct what someone would have seen
at some point in time. If the view has been updated and rolled back, this will
show it. The struct has the following fields:
+
+| Required/Optional | Field Name | Description |
+|-------------------|------------|-------------|
+| Required | timestamp-ms | The timestamp when the referenced version was made
the current version |
+| Required | version-id | Version id of the view |
+
+#### Summary
+
+Field “summary” is a string map with the following keys. Only `operation` is
required. Engines may store additional key-value pairs in this map.
+
+| Required/Optional | Key | Value |
+|-------------------|-----|-------|
+| Required | operation | A string value indicating the view operation that
caused this metadata to be created. Allowed values are “create” and “replace”. |
+| Optional | engine-version | A string value indicating the version of the
engine that performed the operation |
+
+#### Representations
+
+Each representation is stored as an object with only one common field "type".
+The rest of the fields are interpreted based on the type.
+There is only one type of representation defined in the spec.
+
+##### Original View Definition in SQL
+
+This type of representation stores the original view definition in SQL and its
SQL dialect.
+
+| Required/Optional | Field Name | Description |
+|-------------------|------------|-------------|
+| Required | type | A string indicating the type of representation. It is set
to "sql" for this type. |
+| Required | sql | A string representing the original view definition in SQL |
+| Required | dialect | A string specifying the dialect of the ‘sql’ field. It
can be used by the engines to detect the SQL dialect. |
+| Optional | schema-id | ID of the view's schema when the version was created |
+| Optional | default-catalog | A string specifying the catalog to use when the
table or view references in the view definition do not contain an explicit
catalog. |
+| Optional | default-namespace | The namespace to use when the table or view
references in the view definition do not contain an explicit namespace. Since
the namespace may contain multiple parts, it is serialized as a list of
strings. |
+| Optional | field-aliases | A list of strings of field aliases optionally
specified in the create view statement. The list should have the same length as
the schema's top level fields. See the example below. |
+| Optional | field-docs | A list of strings of field comments optionally
specified in the create view statement. The list should have the same length as
the schema's top level fields. See the example below. |
+
+For `CREATE VIEW v (alias_name COMMENT 'docs', alias_name2, ...) AS SELECT
col1, col2, ...`,
+the field aliases are 'alias_name', 'alias_name2', and etc., and the field
docs are 'docs', null, and etc.
+
+## Appendix A: An Example
+
+The JSON metadata file format is described using an example below.
+
+Imagine the following sequence of operations:
+
+* `CREATE TABLE base_tab(c1 int, c2 varchar);`
+* `INSERT INTO base_tab VALUES (1,’one’), (2,’two’);`
+* `CREATE VIEW common_view AS SELECT * FROM base_tab;`
+* `CREATE OR REPLACE VIEW common_view AS SELECT count(*) AS my_cnt FROM
base_tab;`
+
+The metadata JSON file created at the end of step 3 looks as follows. The file
path looks like:
+`s3://my_company/my/warehouse/anorwood.db/common_view`
+
+The path is intentionally similar to the path for iceberg tables and contains
a ‘metadata’ directory.
(`METASTORE_WAREHOUSE_DIR/<dbname>.db/<viewname>/metadata`)
+
+The metadata directory contains View Version Metadata files. The text after
'=>' symbols describes the fields.
+```
+{
+ "format-version" : 1, => JSON format. Will change as format evolves.
+ "location" : "s3n://my_company/my/warehouse/anorwood.db/common_view",
+ "current-version-id" : 1, => current / latest version of the view. ‘1’ here
since this metadata was created when the view was created.
+ "properties" : { => shows properties of the view
+ "comment" : "View captures all the data from the table" => View comment
+ },
+ "versions" : [ { => Last few versions of the view.
+ "version-id" : 1,
+ "parent-version-id" : -1,
+ "timestamp-ms" : 1573518431292,
+ "summary" : {
+ "operation" : "create", => View operation that caused this metadata to
be created
+ "engineVersion" : "presto-350", => Version of the engine that performed
the operation (create / replace)
+ },
+ "representations" : [ { => SQL metadata of the view
+ "type" : "sql",
+ "sql" : "SELECT *\nFROM\n base_tab\n", => original view SQL
+ "dialect" : "presto",
+ "schema-id" : 1,
+ "default-catalog" : "iceberg",
+ "default-namespace" : [ "anorwood" ]
+ } ],
+ } ],
+ "version-log" : [ { => Log of the created versions
+ "timestamp-ms" : 1573518431292,
+ "version-id" : 1
+ } ],
+ "schemas": [ { => Schema of the view expressed in Iceberg types
+ "schema-id": 1,
+ "type" : "struct",
+ "fields" : [ {
+ "id" : 0,
+ "name" : "c1",
+ "required" : false,
+ "type" : "int",
+ "doc" : "" => Column comment
+ }, {
+ "id" : 1,
+ "name" : "c2",
+ "required" : false,
+ "type" : "string",
+ "doc" : ""
+ } ]
+ } ],
+ "current-schema-id": 1
+}
+```
+
+The Iceberg / view library creates a new metadata JSON file every time the
view undergoes a DDL change. This way the history of how the view evolved can
be maintained. Following metadata JSON file was created at the end of Step 4.
+
+```
+{
+ "format-version" : 1,
+ "location" : "s3n://my_company/my/warehouse/anorwood.db/common_view",
+ "current-version-id" : 2,
+ "properties" : { => shows properties of the view
+ "comment" : "View captures count of the data from the table"
+ },
+ "versions" : [ {
+ "version-id" : 1,
+ "parent-version-id" : -1,
+ "timestamp-ms" : 1573518431292,
+ "summary" : {
+ "operation" : "create",
+ "engineVersion" : "presto-350",
+ },
+ "representations" : [ {
+ "type" : "sql",
+ "sql" : "SELECT *\nFROM\n base_tab\n",
+ "dialect" : "presto",
+ "schema-id" : 1,
+ "default-catalog" : "iceberg",
+ "default-namespace" : [ "anorwood" ]
+ } ],
+ "properties" : { }
+ }, {
+ "version-id" : 2,
+ "parent-version-id" : 1, => Version 2 was created on top of version 1,
making parent-version-id 1
+ "timestamp-ms" : 1573518440265,
+ "summary" : {
+ "operation" : "replace", => The ‘replace’ operation caused this latest
version creation
+ "engineVersion" : "spark-2.4.4",
+ },
+ "representations" : [ {
+ "type" : "sql",
+ "sql" : "SELECT \"count\"(*) my_cnt\nFROM\n base_tab\n", => Note the
updated text from the ‘replace’ view statement
+ "dialect" : "spark",
+ "schema-id" : 2,
+ "default-catalog" : "iceberg",
+ "default-namespace" : [ "anorwood" ]
+ },
+ } ],
+ "version-log" : [ {
+ "timestamp-ms" : 1573518431292,
+ "version-id" : 1
+ }, {
+ "timestamp-ms" : 1573518440265,
+ "version-id" : 2
+ } ],
+ "schemas": [ { => Schema of the view expressed in Iceberg types
+ "schema-id": 1,
+ "type" : "struct",
+ "fields" : [ {
+ "id" : 0,
+ "name" : "c1",
+ "required" : false,
+ "type" : "int",
+ "doc" : "" => Column comment
+ }, {
+ "id" : 1,
+ "name" : "c2",
+ "required" : false,
+ "type" : "string",
+ "doc" : ""
+ } ]
+ }, { => Schema change is reflected here
+ "schema-id": 2,
+ "type" : "struct",
+ "fields" : [ {
+ "id" : 0,
+ "name" : "my_cnt",
+ "required" : false,
+ "type" : "long",
+ "doc" : ""
+ } ]
+ } ],
+ "current-schema-id": 2
+}
+```
