Hi Micah, Thanks for taking the time to review and giving your feedback here. Responding:
> 1. There are a lot of idiosyncrasies to getting the types systems/structures lined up for any given serialization format to support shredding (e.g. in protobuf, how to handle recursive schemas and well known types can have multiple reasonable interpretations). Agreed, and we are aware of these. The type itself does no shredding and interprets nothing; it keeps the bytes as the value, so recursion, well-known types, and Any never have to be lined up with Parquet. For projection, anything that does not map cleanly stays in the bytes and is read by decoding rather than being forced into a column. We have this working in a prototype and will lay out how each case is handled in the write-up. > 2. If we expect parquet libraries to do the shredding, it means taking on additional dependencies or maintaining parsing libraries or each implementation. No, the Parquet library should have to parse. The type never decodes the payload, so reading it is the bytes plus the two parameters. Shredding and projection stay on the engine side, so the parser dependency lives in the engine and not in any Parquet library. Originally I had this in the parquet implementation locally, but I can minimize it in the POC and figure out the right boundary for the engine. > 3. I don't think there are really any engines that would support querying data in the form today. Do you have proposals open against any OSS engines to support this type? Correct, nothing queries this today, and we have no proposal open against an OSS engine. The adjacent demand is real: engines are already building projection and pushdown into encoded payload columns (Iceberg #16064 <https://github.com/apache/iceberg/issues/16064>, DataFusion #21308 <https://github.com/apache/datafusion/issues/21308>, Spark #58303 <https://github.com/apache/spark/pull/58303>), all on Variant, which re-encodes the bytes and cannot return the original message. That is the gap this type fills where the exact bytes are the system of record, such as audit, replay, lineage, and regulated data. We can prototype on one engine and can bring it as a reference to open the discussion. > 4. More of a bikeshedding comment but "WIRE" might not be the best name for the type (suggested would be SERIALIZED_VALUE). Not attached to the name. Can go with SERIALIZED_VALUE since it is explicit or open it up to the community for a different name. > IMO, I think a reasonable path forward might be to separate this into two components: Yes. FORMAT as a free-form string with a recommended list (PROTO, THRIFT, JSON, BSON). The schema identity is the part that needs the most thought, and we will work it through in the write-up. > 2. Have a separate proposal on how to annotate projections on SERIALIZED_VALUE columns in parquet. This would track a segment of the schema as a projection of a SERIALIZED_VALUE column. This would allow flexibility because we could use a variant shredding style for schemaless serialization formats and the more natural structured shredding for values with a schema. Agreed, that split is the right one: structured shredding for schema-bearing formats and a Variant-shredding style for schemaless ones. One of the reasons I kept it aligned with Variant shredding since it was established. We can spell it out in a separate proposal. We will write up both proposals and update this thread. On Fri, Sep 25, 2026 at 2:17 PM Micah Kornfield <[email protected]> wrote: > Hi Neelesh, > Thanks for the proposal. While I think being able to allow annotating > bytes or string columns as some sort of encoded structure I have a few > concerns with the proposal: > > 1. There are a lot of idiosyncrasies to getting the types > systems/structures lined up for any given serialization format to support > shredding (e.g. in protobuf, how to handle recursive schemas and well known > types can have multiple reasonable interpretations). > 2. If we expect parquet libraries to do the shredding, it means taking on > additional dependencies or maintaining parsing libraries or each > implementation. > 3. I don't think there are really any engines that would support querying > data in the form today. Do you have proposals open against any OSS engines > to support this type? > 4. More of a bikeshedding comment but "WIRE" might not be the best name > for the type (suggested would be SERIALIZED_VALUE). > > IMO, I think a reasonable path forward might be to separate this into two > components: > 1. Add a logical type that designates a column as a SERIALIZED_VALUE. > This should probably hold two key elements: > a. FORMAT (I think free-form string with standardized list is fine for > this. e.g. PROTO, THRIFT) > b. SCHEMA: This needs some more thought but details on how the schema > could be retrieved (whether stored inline or a registry, etc). Also if it > lacks a schema we should also capture this information (e.g. this could > subsume existing JSON/BSON types today). > 2. Have a separate proposal on how to annotate projections on > SERIALIZED_VALUE columns in parquet. This would track a segment of the > schema as a projection of a SERIALIZED_VALUE column. This would allow > flexibility because we could use a variant shredding style for schemaless > serialization formats and the more natural structured shredding for values > with a schema. > > Hopefully with these two components we could push the business logic of the > projection to engines leading to a more maintainable state. Before we went > too far with #2, I would like to see evidence that an engine would actually > support using the projections. > > Hope this helps. > > Thanks, > -Micah > > On Fri, Sep 11, 2026 at 3:12 PM Neelesh Salian <[email protected]> > wrote: > > > Hi all, > > > > I have written up a proposal for a new Parquet logical type, tentatively > > called WIRE. > > > > The Proposal document: > > > > > https://docs.google.com/document/d/1bt5YAeB0dvRT6hVt8bkw2AM3D5EyThuyvtLxQBHpG5s/edit?usp=sharing > > > > > > Serialized messages, protobuf especially, are a common way data is stored > > at rest. There are a few ways to put them in Parquet today, and each one > > costs you something: > > > > 1. Store the raw bytes as a blob. You cannot query anything inside them. > > 2. Explode every field into columns (what parquet-protobuf and > > parquet-thrift do). You pay a full decode on every write, a migration on > > every schema change, > > and you drop what does not fit a relational schema (unknown fields, > oneof, > > byte-exactness). > > 3. Convert to Variant. You re-encode the message and carry a per-row > > field-name dictionary that protobuf never needed. > > > > So today you either cannot query the data, or you pay a lot and lose > > fidelity to make it queryable. > > > > WIRE keeps the message as it is and makes the fields you query columnar. > It > > stores the message verbatim in a "value" column, so the original is > > recoverable byte for byte and there is no re-encode or migration when the > > schema grows. > > The fields you actually query are shredded into native Parquet columns > with > > real statistics and pushdown, reusing the Variant shredding layout. > > There is no metadata column, because protobuf and thrift address fields > by > > number, so there is no name dictionary to carry, unlike Variant. > > A reader that does not know WIRE just sees plain columns. > > The first version covers protobuf and thrift compact, and the design is > > encoding-general, so more can follow. > > > > I would appreciate feedback on two things: > > > > 1. The design, and whether this is worth bringing forward as a formal > > proposal. If so, I can follow the proposal lifecycle and add it to the > > Active Proposals list. > > 2. What is needed for cross-implementation verification? With Variant, it > > came down to two implementations proven to read and write each other's > > files against shared fixtures in parquet-testing, and I would like to > plan > > for that early. > > > > If there is interest, I plan to build the parquet-java reference > > implementation and drive the spec work, and would help stand up the > > cross-language verification. > > > > Looking forward to your thoughts. > > > > Thanks, > > Neelesh > > >
