Hi folks, I've opened PRs for parquet-format [1] and parquet-java [2] to support the FLBA(12) physical type for the TimestampType logical annotation. Please take a look if interested and comment.
Best, Div Arora [1] https://github.com/apache/parquet-format/pull/601 [2] https://github.com/apache/parquet-java/pull/3680 On Thu, Jul 16, 2026 at 4:16 PM Russell Spitzer <[email protected]> wrote: > I'd lean toward choosing a bit width and going forward for similar reasons. > I don't think a lot > of engines/users are going to custom-define their bit-width ahead of time. > It seems more > likely if we wanted to benefit readers, we would just store the max > used-bit width as a stat. > > On Thu, Jul 16, 2026 at 12:30 AM Micah Kornfield <[email protected]> > wrote: > > > Being able to use compact data types is a good thing, no? :-) > > > > > > Generally yes, but I think there is a difference here between Decimal and > > Timestamp. Decimal a user is specifying precision and scale up front, so > we > > can calculate the exact FLBA width. The closest thing we have with > > timestamp is for SQL purposes it has to cover a 10K year range. But in > > general, we won't know the exact min and max value up front. > > > > For existing granularities of milliseconds and microseconds there is not > > any benefit to using FLBA (users should use existing physical integer > > values). Extra engineering effort for these granularities doesn't seem > > worthwhile. > > > > For nanoseconds, I believe we could use the FLBA<8> type. However, some > > systems are starting to move toward picoseconds [1][2] (and I've heard > > femtoseconds discussed) so we will eventually want to support all of > those > > ranges. FLBA<9> allows us to do it all with the same code. > > > > I think extending the spec to support different size types is certainly > > something we could do in the future, but doing it now seems like it is > just > > more code to maintain without a concrete use-case. > > > > Cheers, > > Micah > > > > [1] > > > > > https://docs.cloud.google.com/bigquery/docs/reference/storage/rpc/google.cloud.bigquery.storage.v1?content_ref=read%20api%20will%20return%20full%20precision%20picosecond%20value%20the%20value%20will%20be%20encoded%20as%20a%20string%20which%20conforms%20to%20iso%208601%20format#picostimestampprecision > > [2] https://trino.io/docs/current/language/types.html#timestamp-p > > > > On Fri, Jul 10, 2026 at 2:38 AM Antoine Pitrou <[email protected]> > wrote: > > > > > > > > Le 10/07/2026 à 11:27, Divjot Arora via dev a écrit : > > > >> Yeah, well, regardless, this is derailing the discussion quite a > bit. > > > > This proposal is not about extending DBP to FLBA. Can we keep it on > > > topic? > > > > > > > > Agreed, let's keep this discussion focused to extended-precision > > > > timestamps. We can have separate discussions on encodings.. > > > > > > > >> And conversely, I do not see what we get out of mandating FLBA<12>. > > :-) > > > > > > > > FLBA<N> works, but I would be concerned about added complexity on the > > > > reader side as I assume engines will want to use compact data types > to > > > > represent timestamps as much as possible, so they'd have to manage > > > > different data types for different values of N. > > > > > > Being able to use compact data types is a good thing, no? :-) > > > > > > The good news is that some readers already have similar code for > reading > > > Decimals from arbitrary FLBA<N>. For example, Parquet C++ can read > > > Decimal-annotated FLBA<N> into Arrow Decimal128 or Decimal256, > depending > > > on the advertised precision. > > > > > > (and the code to do that is reasonably simple: you're really > > > sign-extending from one integer width to another) > > > > > > Regards > > > > > > Antoine. > > > > > > > > > > > >
