nssalian commented on code in PR #17537: URL: https://github.com/apache/iceberg/pull/17537#discussion_r3763743252
########## site/docs/blog/posts/2026-08-10-variant-in-apache-iceberg.md: ########## @@ -0,0 +1,144 @@ +--- +date: 2026-08-10 +title: "Semi-Structured Data in Apache Iceberg: Meet the Variant Type" +slug: variant-in-apache-iceberg +authors: + - nssalian +categories: + - blog +--- + +<!-- + - Licensed to the Apache Software Foundation (ASF) under one or more + - contributor license agreements. See the NOTICE file distributed with + - this work for additional information regarding copyright ownership. + - The ASF licenses this file to You under the Apache License, Version 2.0 + - (the "License"); you may not use this file except in compliance with + - the License. You may obtain a copy of the License at + - + - http://www.apache.org/licenses/LICENSE-2.0 + - + - Unless required by applicable law or agreed to in writing, software + - distributed under the License is distributed on an "AS IS" BASIS, + - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + - See the License for the specific language governing permissions and + - limitations under the License. + --> + +Semi-structured data, such as JSON-like documents whose fields differ from row to row, has always been a poor fit for table formats built around fixed schemas. Iceberg v3 adds the Variant type for exactly this data: a single column can hold values of arbitrary, evolving shape, stored in a compact binary form that engines read and write consistently. + +This is the first post in a series on Variant in Apache Iceberg. It covers what Variant is, why it exists, and how it fits into an Iceberg table. Later posts cover shredding, the technique that stores frequently accessed Variant fields as typed, columnar data. Variant is stored in Parquet, Avro, and ORC; shredding is currently available only in Parquet. + +<!-- more --> + +## What is Variant? + +Variant is a self-describing type for semi-structured data. A single Variant column can hold objects, arrays, and primitive values. + +## Why Variant? + +Consider event data whose shape changes over time: + +```json +{"timestamp": "2026-01-15T10:30:00Z", "user": 5, "event": "login"} +{"timestamp": "2026-01-15T11:45:00Z", "user": 5, "event": "purchase", "amount": 99.99} +{"timestamp": "2026-01-15T12:00:00Z", "user": 7, "event": "login", "device": "mobile"} +``` + +Two traditional approaches handle this, and both have drawbacks: + +- **JSON stored as a string.** This is flexible, but reading a single field means parsing the whole text. JSON's type system is also thin: a timestamp is just a string, and a number's precision is ambiguous. +- **A rigid, flattened schema.** This is fast to query, but every new field is a schema migration, and sparse or one-off fields waste space. + +Binary encodings such as BSON store values in binary form, but they still repeat every field name (`"timestamp"`, `"user"`, `"event"`) in every row. + +Variant is as flexible as JSON but stores data in a compact, typed binary form. Values keep their native types: a timestamp stays a timestamp and a decimal keeps its precision, instead of collapsing to JSON's strings and numbers. Field names are stored once in binary, so there is none of the per-row string overhead of JSON or BSON. No schema is declared up front, so documents of different shapes coexist in one column and a new field needs no migration. Review Comment: You're right. There's per-row overhead, the name dictionary sits in every value's metadata. The only saving is within a value, where a repeated name is referenced by id instead of re-inlined. I cut the "stored once" line and the BSON bit to simplify. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
