Agree with Steven that data file format change needs a new discussion thread. Also, my preference is to converge on one file format in the Java reference implementation while leaving the options open in the Spec.
On Tue, Oct 20, 2026 at 4:43 AM Steven Wu <[email protected]> wrote: > Kevin, it is probably better to start a new thread for proposing a format > change for data files. > > On Mon, Oct 19, 2026 at 1:25 PM Kevin Liu <[email protected]> wrote: > >> >> Bumping this thread. Now that the AMT spec is merged [1], V4 manifests >> are written in Parquet. Tables upgraded from earlier versions can keep >> their existing Avro manifests, but any new or rewritten manifest has to be >> a V4 (Parquet) manifest. >> >> I want to pick back up what Russell and Manu brought up earlier about >> converging on a single file format. With V4 manifests now Parquet-only, I >> think we should do the same for data files. >> >> >> >> *I'd like to propose dropping ORC and Avro as data file formats in V4.*I >> looked into where ORC and Avro support is today, and I think there's a >> pretty good case for it. There are gaps in V3 feature support, most of the >> non-Java implementations don't support them, and there's not much happening >> upstream in ORC or Avro on the features we need. >> >> What's the best next step here? Happy to share more details in this >> thread or start a separate one. >> >> Best, >> Kevin Liu >> >> [1] https://github.com/apache/iceberg/pull/16025 >> >> On Thu, Aug 13, 2026 at 1:09 AM Péter Váry <[email protected]> >> wrote: >> >>> The most important thing, in my view, is that we retain InternalData >>> (thank Ryan creating this abstraction). We should preserve the flexibility >>> to change course in the future if needed. >>> >>> For that reason, I have a slight preference for continuing to support >>> multiple metadata file formats as well. Having more than one supported >>> format helps validate that our APIs are truly general and not overly >>> tailored to a single implementation. >>> >>> That said, I'm more concerned about data file formats than metadata file >>> formats. Given the additional maintenance effort involved and the >>> relatively smaller surface area, I would not oppose removing Avro as a >>> metadata format if that is what the community, especially those doing the >>> implementation work, prefers. >>> >>> Manu Zhang <[email protected]> ezt írta (időpont: 2026. aug. 13., >>> Cs, 4:42): >>> >>>> > I want to clarify that the purpose of this thread is only about >>>> manifest files. >>>> Of course, Steven. I'm just echoing Russell's comments regarding the >>>> possibility of a single file format. >>>> >>>> > I think that, for both metadata and data files, we should keep the >>>> infrastructure open to experimenting with and if we decide so, adopting new >>>> file formats. >>>> Hi Peter, I don't think we are precluding that possibility, but we are >>>> focusing on one manifests file format for the reference implementation. We >>>> can maintain format flexibility through robust abstractions and interfaces, >>>> similar to the great File Format API you contributed. >>>> >>>> >>>> On Thu, Aug 13, 2026 at 3:19 AM Steven Wu <[email protected]> wrote: >>>> >>>>> > I recall a separate discussion in our community sync meetings about >>>>> deprecating ORC as well >>>>> >>>>> I want to clarify that the purpose of this thread is only about >>>>> manifest files. I like to keep the data file formats discussion separate >>>>> (adding new or deprecating old). >>>>> >>>>> In v1-v3, manifest files are only in Avro format. As Anoop said, an >>>>> upgraded v4 table can still contain pre-v4 Avro manifest files and read >>>>> support for them is still needed. Here, we propose limiting v4 >>>>> manifest files to Parquet format only, due to the benefits mentioned in >>>>> this thread. >>>>> >>>>> On Wed, Aug 12, 2026 at 7:02 AM Péter Váry < >>>>> [email protected]> wrote: >>>>> >>>>>> I think that, for both metadata and data files, we should keep the >>>>>> infrastructure open to experimenting with and if we decide so, adopting >>>>>> new >>>>>> file formats. File formats are a foundational layer of the ecosystem, and >>>>>> tying ourselves to a single downstream project creates unnecessary risk. >>>>>> >>>>>> If that project is unwilling to accept features that are critical for >>>>>> our use cases, or slow to adopt new technological advances, or even >>>>>> abandoned, our ability to evolve would be constrained by decisions >>>>>> outside >>>>>> our control. Maintaining format flexibility helps preserve innovation, >>>>>> interoperability, and long-term resilience of Iceberg. >>>>>> >>>>>> Manu Zhang <[email protected]> ezt írta (időpont: 2026. aug. >>>>>> 12., Sze, 5:20): >>>>>> >>>>>>> +1. I do not see any downside to making V4 manifests Parquet-only. >>>>>>> Agree with Russell that we should converge on a single file format if >>>>>>> possible (I recall a separate discussion in our community sync meetings >>>>>>> about deprecating ORC as well). >>>>>>> >>>>>>> Thanks, >>>>>>> Manu >>>>>>> >>>>>>> >>>>>>> On Wed, Aug 12, 2026 at 3:30 AM Russell Spitzer < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> I'm just pretty neutral here with a slight bias towards Parquet >>>>>>>> only. I don't think supporting Avro is much more difficult, but that's >>>>>>>> mostly my prejudice from the Java library. It's probably better for us >>>>>>>> to >>>>>>>> converge on a single file format if possible so I think Parquet only >>>>>>>> is a >>>>>>>> good step in that direction. >>>>>>>> >>>>>>>> On Tue, Aug 11, 2026 at 1:42 PM Anoop Johnson <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> It would be great to make V4 manifest Parquet-only. This >>>>>>>>> simplifies the spec and the implementation complexity and should >>>>>>>>> improve >>>>>>>>> read performance as Steven noted. Also we don't currently keep track >>>>>>>>> of the >>>>>>>>> root manifest format currently. So if we really want to support Avro >>>>>>>>> root >>>>>>>>> manifests, we need to start tracking the format. The alternative is to >>>>>>>>> require writers to always produce the root manifest in Parquet. But >>>>>>>>> then >>>>>>>>> there is no point in supporting Avro leaf manifests if writers are >>>>>>>>> required >>>>>>>>> to produce root in Parquet anyway. >>>>>>>>> >>>>>>>>> Note that for v4 table upgrades we will support v3 leaf manifests >>>>>>>>> which are in Avro - so the Parquet restriction applies to new V4 >>>>>>>>> manifests >>>>>>>>> only. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Anoop >>>>>>>>> >>>>>>>>> On Tue, Aug 11, 2026 at 11:29 AM Steven Wu <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> During today’s column update sync, we discussed whether to keep >>>>>>>>>> the Avro format option for V4 manifest files. The initial >>>>>>>>>> inclination was >>>>>>>>>> to keep it because it already exists, even though Avro won’t support >>>>>>>>>> projection reads on manifest files (including column stats). >>>>>>>>>> >>>>>>>>>> With both Avro and Parquet available, integrations and engines >>>>>>>>>> would need to choose. We expect most will pick Parquet for >>>>>>>>>> projection-read >>>>>>>>>> support. That raises a question: should we instead require Parquet >>>>>>>>>> for V4 >>>>>>>>>> manifests to reduce the cognitive and decision burden on >>>>>>>>>> integrations? >>>>>>>>>> >>>>>>>>>> This also aligns with Iceberg’s usual priority of read >>>>>>>>>> performance—especially query scan planning, where projecting column >>>>>>>>>> stats >>>>>>>>>> from manifests matters. >>>>>>>>>> >>>>>>>>>> In the sync, most people generally agreed with making V4 >>>>>>>>>> manifests *Parquet-only*. I’m sending this to collect any >>>>>>>>>> feedback or concerns from the broader community before we settle on >>>>>>>>>> that >>>>>>>>>> direction. >>>>>>>>>> Thanks, >>>>>>>>>> Steven >>>>>>>>>> >>>>>>>>>> P.S. I uploaded the recording to the YouTube channel here >>>>>>>>>> <https://youtu.be/dDZKzz3PKIk?t=2925> (relevant discussion >>>>>>>>>> started at 48:45). >>>>>>>>>> >>>>>>>>>
