Kevin, it is probably better to start a new thread for proposing a format change for data files.
On Mon, Oct 19, 2026 at 1:25 PM Kevin Liu <[email protected]> wrote: > > Bumping this thread. Now that the AMT spec is merged [1], V4 manifests are > written in Parquet. Tables upgraded from earlier versions can keep their > existing Avro manifests, but any new or rewritten manifest has to be a V4 > (Parquet) manifest. > > I want to pick back up what Russell and Manu brought up earlier about > converging on a single file format. With V4 manifests now Parquet-only, I > think we should do the same for data files. > > > > *I'd like to propose dropping ORC and Avro as data file formats in V4.*I > looked into where ORC and Avro support is today, and I think there's a > pretty good case for it. There are gaps in V3 feature support, most of the > non-Java implementations don't support them, and there's not much happening > upstream in ORC or Avro on the features we need. > > What's the best next step here? Happy to share more details in this thread > or start a separate one. > > Best, > Kevin Liu > > [1] https://github.com/apache/iceberg/pull/16025 > > On Thu, Aug 13, 2026 at 1:09 AM Péter Váry <[email protected]> > wrote: > >> The most important thing, in my view, is that we retain InternalData >> (thank Ryan creating this abstraction). We should preserve the flexibility >> to change course in the future if needed. >> >> For that reason, I have a slight preference for continuing to support >> multiple metadata file formats as well. Having more than one supported >> format helps validate that our APIs are truly general and not overly >> tailored to a single implementation. >> >> That said, I'm more concerned about data file formats than metadata file >> formats. Given the additional maintenance effort involved and the >> relatively smaller surface area, I would not oppose removing Avro as a >> metadata format if that is what the community, especially those doing the >> implementation work, prefers. >> >> Manu Zhang <[email protected]> ezt írta (időpont: 2026. aug. 13., >> Cs, 4:42): >> >>> > I want to clarify that the purpose of this thread is only about >>> manifest files. >>> Of course, Steven. I'm just echoing Russell's comments regarding the >>> possibility of a single file format. >>> >>> > I think that, for both metadata and data files, we should keep the >>> infrastructure open to experimenting with and if we decide so, adopting new >>> file formats. >>> Hi Peter, I don't think we are precluding that possibility, but we are >>> focusing on one manifests file format for the reference implementation. We >>> can maintain format flexibility through robust abstractions and interfaces, >>> similar to the great File Format API you contributed. >>> >>> >>> On Thu, Aug 13, 2026 at 3:19 AM Steven Wu <[email protected]> wrote: >>> >>>> > I recall a separate discussion in our community sync meetings about >>>> deprecating ORC as well >>>> >>>> I want to clarify that the purpose of this thread is only about >>>> manifest files. I like to keep the data file formats discussion separate >>>> (adding new or deprecating old). >>>> >>>> In v1-v3, manifest files are only in Avro format. As Anoop said, an >>>> upgraded v4 table can still contain pre-v4 Avro manifest files and read >>>> support for them is still needed. Here, we propose limiting v4 >>>> manifest files to Parquet format only, due to the benefits mentioned in >>>> this thread. >>>> >>>> On Wed, Aug 12, 2026 at 7:02 AM Péter Váry <[email protected]> >>>> wrote: >>>> >>>>> I think that, for both metadata and data files, we should keep the >>>>> infrastructure open to experimenting with and if we decide so, adopting >>>>> new >>>>> file formats. File formats are a foundational layer of the ecosystem, and >>>>> tying ourselves to a single downstream project creates unnecessary risk. >>>>> >>>>> If that project is unwilling to accept features that are critical for >>>>> our use cases, or slow to adopt new technological advances, or even >>>>> abandoned, our ability to evolve would be constrained by decisions outside >>>>> our control. Maintaining format flexibility helps preserve innovation, >>>>> interoperability, and long-term resilience of Iceberg. >>>>> >>>>> Manu Zhang <[email protected]> ezt írta (időpont: 2026. aug. >>>>> 12., Sze, 5:20): >>>>> >>>>>> +1. I do not see any downside to making V4 manifests Parquet-only. >>>>>> Agree with Russell that we should converge on a single file format if >>>>>> possible (I recall a separate discussion in our community sync meetings >>>>>> about deprecating ORC as well). >>>>>> >>>>>> Thanks, >>>>>> Manu >>>>>> >>>>>> >>>>>> On Wed, Aug 12, 2026 at 3:30 AM Russell Spitzer < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> I'm just pretty neutral here with a slight bias towards Parquet >>>>>>> only. I don't think supporting Avro is much more difficult, but that's >>>>>>> mostly my prejudice from the Java library. It's probably better for us >>>>>>> to >>>>>>> converge on a single file format if possible so I think Parquet only is >>>>>>> a >>>>>>> good step in that direction. >>>>>>> >>>>>>> On Tue, Aug 11, 2026 at 1:42 PM Anoop Johnson <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> It would be great to make V4 manifest Parquet-only. This simplifies >>>>>>>> the spec and the implementation complexity and should improve read >>>>>>>> performance as Steven noted. Also we don't currently keep track of the >>>>>>>> root >>>>>>>> manifest format currently. So if we really want to support Avro root >>>>>>>> manifests, we need to start tracking the format. The alternative is to >>>>>>>> require writers to always produce the root manifest in Parquet. But >>>>>>>> then >>>>>>>> there is no point in supporting Avro leaf manifests if writers are >>>>>>>> required >>>>>>>> to produce root in Parquet anyway. >>>>>>>> >>>>>>>> Note that for v4 table upgrades we will support v3 leaf manifests >>>>>>>> which are in Avro - so the Parquet restriction applies to new V4 >>>>>>>> manifests >>>>>>>> only. >>>>>>>> >>>>>>>> Best, >>>>>>>> Anoop >>>>>>>> >>>>>>>> On Tue, Aug 11, 2026 at 11:29 AM Steven Wu <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> During today’s column update sync, we discussed whether to keep >>>>>>>>> the Avro format option for V4 manifest files. The initial inclination >>>>>>>>> was >>>>>>>>> to keep it because it already exists, even though Avro won’t support >>>>>>>>> projection reads on manifest files (including column stats). >>>>>>>>> >>>>>>>>> With both Avro and Parquet available, integrations and engines >>>>>>>>> would need to choose. We expect most will pick Parquet for >>>>>>>>> projection-read >>>>>>>>> support. That raises a question: should we instead require Parquet >>>>>>>>> for V4 >>>>>>>>> manifests to reduce the cognitive and decision burden on integrations? >>>>>>>>> >>>>>>>>> This also aligns with Iceberg’s usual priority of read >>>>>>>>> performance—especially query scan planning, where projecting column >>>>>>>>> stats >>>>>>>>> from manifests matters. >>>>>>>>> >>>>>>>>> In the sync, most people generally agreed with making V4 manifests >>>>>>>>> *Parquet-only*. I’m sending this to collect any feedback or >>>>>>>>> concerns from the broader community before we settle on that >>>>>>>>> direction. >>>>>>>>> Thanks, >>>>>>>>> Steven >>>>>>>>> >>>>>>>>> P.S. I uploaded the recording to the YouTube channel here >>>>>>>>> <https://youtu.be/dDZKzz3PKIk?t=2925> (relevant discussion >>>>>>>>> started at 48:45). >>>>>>>>> >>>>>>>>
