Hey Everyone,
At the last Iceberg Community Sync, I raised the topic of opening a discussion on how we want to scope v4 to arrive at a shared understanding of what we want to land in the next table spec version. Anyone who regularly attends conferences, meetups, or syncs is likely familiar with specific efforts currently in flight, but as a community, we haven't defined what we believe constitutes a cohesive set of capabilities to target for the next release.
It's early enough at this point to start a discussion and still have time to include critical features not yet identified or push back anything that may either be underdeveloped or poorly understood.
Broadly speaking, the various ongoing workstreams fall into three main categories:
Content Metadata Updates
- Adaptive Metadata Tree (AMT) / Single File Commits
- Column Statistics
- Relative Paths
- Column Append
Table Features
- Check Constraints
- Default Value Expressions
- Generated Columns
Data Types
- File Type
- Vector Type
Some have only recently been proposed (e.g. Vector Type and Generated Columns), while others have been discussed for over a year and are well into implementation (e.g. AMT and Stats).
While many non-table format discussions and efforts are ongoing (MVs, Indexing, REST), focusing attention on what aligns with the v4 release and communicating the scope and progress of that effort would be helpful. For the last release, we used a combination of GitHub milestones and projects (some exist for v4 efforts but are currently incomplete/unmaintained), which I believe many found helpful.
I'd like to hear everyone's thoughts:
- Does this algin with everyone's interests in v4?
- Are there features we've discussed that should/shouldn't be included?
- Do we want to start tracking progress across these efforts?
Looking forward to discussion and feedback,
-Dan
