Hi Neelesh,
Thanks for the proposal. While I think being able to allow annotating
bytes or string columns as some sort of encoded structure I have a few
concerns with the proposal:
1. There are a lot of idiosyncrasies to getting the types
systems/structures lined up for any given serialization format to support
shredding (e.g. in protobuf, how to handle recursive schemas and well known
types can have multiple reasonable interpretations).
2. If we expect parquet libraries to do the shredding, it means taking on
additional dependencies or maintaining parsing libraries or each
implementation.
3. I don't think there are really any engines that would support querying
data in the form today. Do you have proposals open against any OSS engines
to support this type?
4. More of a bikeshedding comment but "WIRE" might not be the best name
for the type (suggested would be SERIALIZED_VALUE).
IMO, I think a reasonable path forward might be to separate this into two
components:
1. Add a logical type that designates a column as a SERIALIZED_VALUE.
This should probably hold two key elements:
a. FORMAT (I think free-form string with standardized list is fine for
this. e.g. PROTO, THRIFT)
b. SCHEMA: This needs some more thought but details on how the schema
could be retrieved (whether stored inline or a registry, etc). Also if it
lacks a schema we should also capture this information (e.g. this could
subsume existing JSON/BSON types today).
2. Have a separate proposal on how to annotate projections on
SERIALIZED_VALUE columns in parquet. This would track a segment of the
schema as a projection of a SERIALIZED_VALUE column. This would allow
flexibility because we could use a variant shredding style for schemaless
serialization formats and the more natural structured shredding for values
with a schema.
Hopefully with these two components we could push the business logic of the
projection to engines leading to a more maintainable state. Before we went
too far with #2, I would like to see evidence that an engine would actually
support using the projections.
Hope this helps.
Thanks,
-Micah
On Fri, Sep 11, 2026 at 3:12 PM Neelesh Salian <[email protected]>
wrote:
> Hi all,
>
> I have written up a proposal for a new Parquet logical type, tentatively
> called WIRE.
>
> The Proposal document:
>
> https://docs.google.com/document/d/1bt5YAeB0dvRT6hVt8bkw2AM3D5EyThuyvtLxQBHpG5s/edit?usp=sharing
>
>
> Serialized messages, protobuf especially, are a common way data is stored
> at rest. There are a few ways to put them in Parquet today, and each one
> costs you something:
>
> 1. Store the raw bytes as a blob. You cannot query anything inside them.
> 2. Explode every field into columns (what parquet-protobuf and
> parquet-thrift do). You pay a full decode on every write, a migration on
> every schema change,
> and you drop what does not fit a relational schema (unknown fields, oneof,
> byte-exactness).
> 3. Convert to Variant. You re-encode the message and carry a per-row
> field-name dictionary that protobuf never needed.
>
> So today you either cannot query the data, or you pay a lot and lose
> fidelity to make it queryable.
>
> WIRE keeps the message as it is and makes the fields you query columnar. It
> stores the message verbatim in a "value" column, so the original is
> recoverable byte for byte and there is no re-encode or migration when the
> schema grows.
> The fields you actually query are shredded into native Parquet columns with
> real statistics and pushdown, reusing the Variant shredding layout.
> There is no metadata column, because protobuf and thrift address fields by
> number, so there is no name dictionary to carry, unlike Variant.
> A reader that does not know WIRE just sees plain columns.
> The first version covers protobuf and thrift compact, and the design is
> encoding-general, so more can follow.
>
> I would appreciate feedback on two things:
>
> 1. The design, and whether this is worth bringing forward as a formal
> proposal. If so, I can follow the proposal lifecycle and add it to the
> Active Proposals list.
> 2. What is needed for cross-implementation verification? With Variant, it
> came down to two implementations proven to read and write each other's
> files against shared fixtures in parquet-testing, and I would like to plan
> for that early.
>
> If there is interest, I plan to build the parquet-java reference
> implementation and drive the spec work, and would help stand up the
> cross-language verification.
>
> Looking forward to your thoughts.
>
> Thanks,
> Neelesh
>
