Hello all, I'm reaching out to help drive this long-running discussion—nearly three years now—towards a final resolution. With Jan's authorization, and my sincere thanks for his sustained effort, I want to help push this issue to the finish line.
To recap, we have two primary proposals on how to handle NaNs in statistics and column indexes: * IEEE 754 Total Order [1]: Proposes adding a new column order IEEE754TotalOrder for FLOAT/DOUBLE/FLOAT16. This provides a defined ordering for every float bit pattern, including NaNs and -0/+0, allowing writers to include NaNs in min/max and removing ambiguity for only-NaN pages. * Combined Approach [2]: Proposes adopting the IEEE 754 total order alongside explicit nan_count(s) fields. This approach mandates the nan_count(s) when the new order is used and clarifies how to handle edge cases from legacy writers. Based on the recent comments, it appears the combined approach [2] is gaining consensus, although the IEEE 754 total order [1] still has strong advocates. I agree with the sentiment that technical direction should be made by consensus, not a vote. To that end, I'd like to solicit further feedback specifically on the combined approach [2] to see if we can achieve the necessary consensus to move forward now. I recall that the total order proposal [1] already has three PoC implementations. For the combined approach [2], I can draft a PoC in parquet-java, but to meet the two-implementation requirement, we would need one more contributor to step up. [1] https://github.com/apache/parquet-format/pull/221 [2] https://github.com/apache/parquet-format/pull/514 Best, Gang On Sat, Aug 16, 2025 at 1:59 AM Gijs Burghoorn <[email protected]> wrote: > > Hello Jan, > > Thank you for pushing this through. Apart from some smaller nits, we also > really like the current proposal. > > Thanks, > Gijs > > On Fri, Aug 15, 2025 at 3:33 PM Andrew Lamb <[email protected]> wrote: > > > I have started organizing a project[1] in arrow-rs 's Parquet reader to try > > and implement this proposal. > > > > Hopefully that can be 1 / 2 open source implementations needed. > > > > Thanks again for helping drive this along, > > Andrew > > > > [1] https://github.com/apache/arrow-rs/issues/8156 > > > > On Wed, Aug 13, 2025 at 5:39 AM Jan Finis <[email protected]> wrote: > > > > > I have now tagged > > > < > > https://github.com/apache/parquet-format/pull/514#issuecomment-3182978173 > > > > > > > the people that argued for total order in the initial PR. Let's see their > > > response. > > > > > > If I understand the adoption process correctly, the next hurdle to > > getting > > > this adopted is two open > > > source (!) implementations proving its feasibility. We already had that > > for > > > IEEE total order. If we > > > prefer the solution with nan counts, we'll need it there as well. I > > myself > > > work on a proprietary > > > implementation, so I'm counting on others here :). Be prepared though, > > this > > > will likely take months > > > unless the interest in this topic has risen to a point where people are > > > eager to jump on the implementation > > > right away. > > > > > > So, I guess it will take some months of soaking time before any formal > > vote > > > can be done > > > (given that we reach consensus that this is what we want and we find > > people > > > for the implementations). > > > > > > Cheers, > > > Jan > > > > > > Am Mi., 13. Aug. 2025 um 01:18 Uhr schrieb Ryan Blue <[email protected]>: > > > > > > > Thanks, Jan. I also went through the combined proposal and it looks > > > mostly > > > > good to me. > > > > > > > > > First of all, to make it quick: Yes, the solution of having > > nan_counts > > > > *and* total order, which was brought up multiple times, does work and > > > > solves more cases than just either of both. > > > > > > > > Great, then we have a solution for both filtering use cases and for > > > moving > > > > ahead with total order. And thanks to Andrew for suggesting this as > > well > > > on > > > > the second PR. I think this also looks like this is something that > > Orson > > > is > > > > okay with given his comments on the latest PR. > > > > > > > > Is there anyone against the combined approach? I don't see a big > > downside > > > > for anyone. It is compatible with previous stats rules, has a NaN > > count, > > > > and allows using either type-specific order or total order. > > > > > > > > Assuming that this satisfies the big objections, I think we should wait > > > for > > > > a few days to make sure everyone has time to check out the new PR and > > > then > > > > vote to adopt it. > > > > > > > > Ryan > > > > > > > > On Mon, Aug 11, 2025 at 6:03 AM Andrew Lamb <[email protected]> > > > > wrote: > > > > > > > > > Thank you Jan -- I read through the new combined proposal, and I > > > thought > > > > it > > > > > looks good and addresses the feedback so far. I left some small style > > > > > suggestions, but nothing that is required from my perspective > > > > > > > > > > > > > > > > > > > > On Sat, Aug 9, 2025 at 9:07 AM Jan Finis <[email protected]> wrote: > > > > > > > > > > > Hey Ryan, > > > > > > > > > > > > Thanks for chiming in. First of all, to make it quick: Yes, the > > > > solution > > > > > of > > > > > > having nan_counts *and* total order, which was brought up multiple > > > > times, > > > > > > does work and solves more cases than just either of both. > > > > > > > > > > > > I strongly prefer continuing to discuss the merits of these > > > approaches > > > > > > > rather than trying to decide with a vote. > > > > > > > > > > > > > > > > > > In theory, I agree that it isn't good to silence a discussion by > > just > > > > > > voting for one possible solution and technical issues should be > > > > > discussed. > > > > > > However, please note that we have been circling on this for over > > two > > > > > years > > > > > > now, including an extended discussion that brought up all arguments > > > > > > multiple times. This is in stark contrast to the > > > > > > speed with which you guys work on the Iceberg spec, for example. > > > There, > > > > > you > > > > > > also do not discuss the merits of various solutions for multiple > > > years. > > > > > You > > > > > > just pick one and merge it after a *reasonable* time of discussion. > > > > > > If you had the speed we currently have here, nothing would get > > done. > > > > > Thus, > > > > > > I see this as a clear case of *"the perfect is the enemy of the > > > good"*. > > > > > > Yes, we can continue looking for the perfect solution, > > > > > > but that will likely lead to keeping us at the status quo, which is > > > the > > > > > > worst of them all. > > > > > > > > > > > > That being said, I'm also happy to create a PR which does both > > total > > > > > order > > > > > > and NaN counts; after all, I just want the issue solved and all > > these > > > > > > solutions are better than the status quo. > > > > > > > > > > > > *As this was now suggest by at least three people, I guess it's > > worth > > > > > > doing, so here you go: > > > > https://github.com/apache/parquet-format/pull/514 > > > > > > <https://github.com/apache/parquet-format/pull/514>* > > > > > > > > > > > > With this, we should have PRs covering most of the solution space. > > > > > > (I'm refusing to create a PR with negative and positive nan_counts; > > > > > > nan_counts + total order has to suffice; the complexity madness has > > > to > > > > > stop > > > > > > somewhere) > > > > > > I still believe that there was an amount of people who already > > found > > > > > > nan_counts too complex and therefore wanted IEEE total order, and > > > these > > > > > > people may not like putting on extra complexity, > > > > > > but let's see, maybe some have also changed their opinion in the > > > > > meantime. > > > > > > > > > > > > > > > > > > *Given all this, we can also first do an informal vote where > > everyone > > > > can > > > > > > vote for which of the three their favorite would be.Maybe a clear > > > > > favorite > > > > > > will emerge and then we can vote on this one.* > > > > > > > > > > > > But of course, we can also take some weeks to discuss the three > > > > > solutions, > > > > > > now that we have PRs for all of them. I just hope this won't make > > us > > > > > > continue for another 2 years, or an > > > > > > infinite stalemate where each solution is vetoed by a PMC member. > > > > > > (Sorry for becoming a bit cynical here; I have just spent way too > > > much > > > > > time > > > > > > of my life with double statistics at this point ;) ...) > > > > > > > > > > > > > > > > > > Cheers, > > > > > > Jan > > > > > > > > > > > > Am Fr., 8. Aug. 2025 um 23:38 Uhr schrieb Ryan Blue < > > > [email protected] > > > > >: > > > > > > > > > > > > > Regarding the process for this, I strongly prefer continuing to > > > > discuss > > > > > > the > > > > > > > merits of these approaches rather than trying to decide with a > > > vote. > > > > I > > > > > > > don't think it is a good practice to use a vote to decide on a > > > > > technical > > > > > > > direction. There are very few situations that warrant it and I > > > don't > > > > > > think > > > > > > > that this is one of them. While this issue has been open for a > > long > > > > > time, > > > > > > > that appears to be the result of it not being anyone's top > > priority > > > > > > rather > > > > > > > than indecision. > > > > > > > > > > > > > > For the technical merits of these approaches, I think that we can > > > > find > > > > > a > > > > > > > middle ground. I agree with Jan that when working with sorted > > > values, > > > > > we > > > > > > > need to know how NaN values were handled and that requires using > > a > > > > > > > well-defined order that includes NaN and its variations (because > > we > > > > > > should > > > > > > > not normalize). Using NaN count is not sufficient for ordering > > > rows. > > > > > > > > > > > > > > Gijs also brings up good points about how NaN values show up in > > > > actual > > > > > > > datasets: not just when used in place of null, but also as the > > > result > > > > > of > > > > > > > normal calculations on abnormal data, like `sqrt(-4.0)` or > > > > `log(-1.0)`. > > > > > > > Both of those present problems when mixed with valid data because > > > of > > > > > the > > > > > > > stats "poisoning" problem, where the range of valid data is > > usable > > > > > until > > > > > > a > > > > > > > single NaN is mixed in. > > > > > > > > > > > > > > Another issue is that NaN is error-prone because "regular" > > > comparison > > > > > is > > > > > > > always false: > > > > > > > ``` > > > > > > > Math.log(-1.0) >= 2 => FALSE > > > > > > > Math.log(-1.0) < 2 => FALSE > > > > > > > 2 > Math.log(-1.0) => FALSE > > > > > > > ``` > > > > > > > > > > > > > > As a result, Iceberg doesn't trust NaN values as either lower or > > > > upper > > > > > > > bounds because we don't want to go back to the code that produced > > > the > > > > > > value > > > > > > > to see what the comparison order was to determine whether NaN > > > values > > > > go > > > > > > > before or after others. > > > > > > > > > > > > > > Total order solves the second issue in theory, but regular > > > comparison > > > > > is > > > > > > > prevalent and not obvious to developers. And it also doesn't help > > > > when > > > > > > NaN > > > > > > > is used instead of null. So using total order is not sufficient > > for > > > > > data > > > > > > > skipping. > > > > > > > > > > > > > > I think the right compromise is to use `min`, `max`, and > > > `nan_count` > > > > > for > > > > > > > data skipping stats (where min and max cannot be NaN) and total > > > > > ordering > > > > > > > for sorting values. That satisfies the data skipping use cases > > and > > > > also > > > > > > > gives us an ordering of unaltered values that we can reason > > about. > > > > > > > > > > > > > > Does anyone think that doesn't work? > > > > > > > > > > > > > > Ryan > > > > > > > > > > > > > > On Fri, Aug 1, 2025 at 8:57 AM Gang Wu <[email protected]> wrote: > > > > > > > > > > > > > > > Thanks Jan for your endless effort on this! > > > > > > > > > > > > > > > > I'm in favor of simplicity and generalism. I think we have > > > already > > > > > > > debated > > > > > > > > a lot > > > > > > > > for `nan_count` in [1] and [2] is the reflection of those > > > > > discussions. > > > > > > > > Therefore > > > > > > > > I am inclined to start a vote for [2] unless there is a > > > > significantly > > > > > > > > better > > > > > > > > proposal. > > > > > > > > > > > > > > > > I would suggest everyone interested in this discussion to > > attend > > > > the > > > > > > > > scheduled > > > > > > > > sync on Aug 6th (detailed below) to spread the word to the > > > broader > > > > > > > > community. > > > > > > > > If we can get a consensus on [2], I can help start the vote and > > > > move > > > > > > > > forward. > > > > > > > > > > > > > > > > *Apache Parquet Community Sync Wednesday, August 6 · 10:00 – > > > > 11:00am > > > > > * > > > > > > > > *Time zone: America/Los_Angeles* > > > > > > > > *Google Meet joining info Video call link: > > > > > > > > https://meet.google.com/bhe-rvan-qjk > > > > > > > > <https://meet.google.com/bhe-rvan-qjk> * > > > > > > > > > > > > > > > > [1] https://github.com/apache/parquet-format/pull/196 > > > > > > > > [2] https://github.com/apache/parquet-format/pull/221 > > > > > > > > > > > > > > > > Best, > > > > > > > > Gang > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 1, 2025 at 6:16 PM Jan Finis <[email protected]> > > > > wrote: > > > > > > > > > > > > > > > > > Hi Gijs, > > > > > > > > > > > > > > > > > > Thank you for bringing up concrete points, I'm happy to > > discuss > > > > > them > > > > > > in > > > > > > > > > detail. > > > > > > > > > > > > > > > > > > NaNs are less common in the SQL world than in the DataFrame > > > world > > > > > > where > > > > > > > > > > NaNs were used for a long time to represent missing values. > > > > > > > > > > > > > > > > > > > > > > > > > > > You could transcode between NULL to NaN before reading and > > > > writing > > > > > to > > > > > > > > > Parquet. You basically mention yourself that NaNs were used > > for > > > > > > missing > > > > > > > > > values, i.e., what is commonly a NULL, which wasn't > > available. > > > > So, > > > > > > > > > semantically, transcoding to NULL would even be the sane > > thing > > > to > > > > > do. > > > > > > > > Yes, > > > > > > > > > that will cost you some cycles, but should be a rather > > > > lightweight > > > > > > > > > operation in comparison to most other operations, so I would > > > > argue > > > > > > that > > > > > > > > it > > > > > > > > > won't totally ruin your performance. Similarly, why should > > > > Parquet > > > > > > play > > > > > > > > > along with a "hack" that was done in other frameworks due to > > > > > > > shortcomings > > > > > > > > > of those frameworks? So from a philosophical point of view, I > > > > think > > > > > > > > > supporting NaNs better is the wrong thing to do. Rather, we > > > > should > > > > > > be a > > > > > > > > > forcing function to align others to better behavior, so > > > appling a > > > > > bit > > > > > > > of > > > > > > > > > force might in the long run make people use NULLs also in > > > > > DataFrames. > > > > > > > > > > > > > > > > > > Of course, your argument also goes into the direction of > > > > > pragmatism: > > > > > > > If a > > > > > > > > > large part of the data science world uses NaNs to encode > > > missing > > > > > > > values, > > > > > > > > > then maybe Parquet should accept this de-facto standard > > rather > > > > than > > > > > > > > > fighting it. That is indeed a valid point. The weight of it > > is > > > > > > > debatable > > > > > > > > > and my personal conclusion is that it's still not worth it, > > as > > > > you > > > > > > can > > > > > > > > > transcode between NULLs and NaNs, but I do agree with its > > > > validity. > > > > > > > > > > > > > > > > > > > > > > > > > > > Since the proposal phrases it as a goal to work "regardless > > of > > > > how > > > > > > they > > > > > > > > > > order NaN w.r.t. other values" this statement feels > > > > out-of-place > > > > > to > > > > > > > me. > > > > > > > > > > Most hardware and most people don't care about total > > ordering > > > > and > > > > > > > > needing > > > > > > > > > > to take it into account while filtering using statistics > > > seems > > > > > like > > > > > > > > > > preferring the special case instead of the common case. > > > Almost > > > > > > noone > > > > > > > > > > filters for specific NaN value bit-patterns. SQL engines > > that > > > > > don't > > > > > > > > have > > > > > > > > > > IEEE total ordering as their default ordering for floats > > will > > > > > also > > > > > > > need > > > > > > > > > to > > > > > > > > > > do more special handling for this. > > > > > > > > > > > > > > > > > > > > > > > > > > > I disagree with the conclusion this statement draws. The > > > current > > > > > > > > behavior, > > > > > > > > > and nan_counts without total ordering, pose a real problem > > > here, > > > > > even > > > > > > > for > > > > > > > > > engines that don't care about bit patterns. I do agree that > > > most > > > > > > > database > > > > > > > > > engines, including the one I'm working on, do not care about > > > bit > > > > > > > patterns > > > > > > > > > and/or sign bits. However, how can our database engine know > > > > whether > > > > > > the > > > > > > > > > writer of a Parquet file saw it the same way? It can't. > > > > Therefore, > > > > > it > > > > > > > > > cannot know whether a writer, for example, ordered NaNs > > before > > > or > > > > > > after > > > > > > > > all > > > > > > > > > other numbers, or maybe ordered them by sign bit. So, if our > > > > > database > > > > > > > > > engine now sees a float column in sorting columns, it cannot > > > > apply > > > > > > any > > > > > > > > > optimization without a lot of special casing, as it doesn't > > > know > > > > > > > whether > > > > > > > > > NaNs will be before all other values, after all other values, > > > or > > > > > > maybe > > > > > > > > > both, depending on sign bit. It could apply contrived logic > > > that > > > > > > tries > > > > > > > to > > > > > > > > > infer where NaNs were placed from the NaN counts of the first > > > and > > > > > > last > > > > > > > > > page, but doing so will be a lot of ugly code that also feels > > > to > > > > be > > > > > > in > > > > > > > > the > > > > > > > > > wrong place. I.e., I don't want to need to load pages or the > > > page > > > > > > > index, > > > > > > > > > just to reason about a sort order. > > > > > > > > > > > > > > > > > > SQL engines that don't have > > > > > > > > > > IEEE total ordering as their default ordering for floats > > will > > > > > also > > > > > > > need > > > > > > > > > to > > > > > > > > > > do more special handling for this. > > > > > > > > > > > > > > > > > > > > > > > > > > > This code, which I would indeed need to write for our engine, > > > is > > > > > > > > comparably > > > > > > > > > trivial. Simply choose the largest possible bit pattern as > > > > > comparison > > > > > > > for > > > > > > > > > upper bounds filtering for NaN, and the smallest possible bit > > > > > pattern > > > > > > > for > > > > > > > > > lower bounds. It's not more than a few lines of code that > > check > > > > > > > whether a > > > > > > > > > filter is NaN and then replace its value with the > > > highest/lowest > > > > > NaN > > > > > > > bit > > > > > > > > > pattern. It is similarly trivial to the special casing I need > > > to > > > > do > > > > > > > with > > > > > > > > > nan_counts, and it is way more trivial than the extra code I > > > > would > > > > > > need > > > > > > > > to > > > > > > > > > write for sorting columns, as depicted above. > > > > > > > > > > > > > > > > > > From a Polars perspective, having a `nan_count` and defining > > > what > > > > > > > > > > happens to the `min` and `max` statistics when a page > > > contains > > > > > only > > > > > > > > NaNs > > > > > > > > > is > > > > > > > > > > enough to allow for all predicate filtering. I think, but > > > > correct > > > > > > me > > > > > > > > if I > > > > > > > > > > am wrong, this is also enough for all SQL engines that > > don't > > > > use > > > > > > > total > > > > > > > > > > ordering. > > > > > > > > > > > > > > > > > > > > > > > > > > > It's not fully enough, as depicted above. Sorting columns > > would > > > > > still > > > > > > > not > > > > > > > > > work properly. > > > > > > > > > > > > > > > > > > As for ways forward, I propose merging the `nan_count` and > > > `sort > > > > > > > > ordering` > > > > > > > > > > proposals into one to make one proposal > > > > > > > > > > > > > > > > > > > > > > > > > > > Note that the initial reason for proposing IEEE total order > > was > > > > > that > > > > > > > > people > > > > > > > > > in the discussion threads found nan_counts to be too complex > > > and > > > > > too > > > > > > > much > > > > > > > > > of an undeserving special case (re-read the discussion in the > > > > > initial > > > > > > > PR > > > > > > > > > <https://github.com/apache/parquet-format/pull/196> to see > > the > > > > > > > > > rationales). > > > > > > > > > So merging both together would go totally against the spirit > > of > > > > why > > > > > > > IEEE > > > > > > > > > total order was proposed. While it has further upsides, the > > > main > > > > > > reason > > > > > > > > was > > > > > > > > > indeed to *not have* nan_counts. If now the proposal would > > even > > > > go > > > > > to > > > > > > > > > positive and negative nan counts (i.e., even more > > complexity), > > > > this > > > > > > > would > > > > > > > > > go 180 degrees into the opposite direction of why people > > wanted > > > > > total > > > > > > > > order > > > > > > > > > in the first place. > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > Jan > > > > > > > > > > > > > > > > > > Am Do., 31. Juli 2025 um 23:23 Uhr schrieb Gijs Burghoorn > > > > > > > > > <[email protected]>: > > > > > > > > > > > > > > > > > > > Hello Jan and others, > > > > > > > > > > > > > > > > > > > > First, let me preface by saying I am quite new here. So I > > > > > apologize > > > > > > > if > > > > > > > > > > there is some other better way to bring up these concerns. > > I > > > > > > > understand > > > > > > > > > it > > > > > > > > > > is very annoying to come in at the 11th hour and start > > > bringing > > > > > up > > > > > > a > > > > > > > > > bunch > > > > > > > > > > of concerns, but I would also like this to be done right. A > > > > > > colleague > > > > > > > > of > > > > > > > > > > mine brought up some concerns and alternative approaches in > > > the > > > > > > > GitHub > > > > > > > > > > thread; I will file some of the concerns here as a > > response. > > > > > > > > > > > > > > > > > > > > > Treating NaNs so specially is giving them attention they > > > > don't > > > > > > > > deserve. > > > > > > > > > > Most data sets do not contain NaNs. If a use case really > > > > requires > > > > > > > them > > > > > > > > > and > > > > > > > > > > needs filtering to ignore them, they can store NULL > > instead, > > > or > > > > > > > encode > > > > > > > > > them > > > > > > > > > > differently. I would prefer the average case over the > > special > > > > > case > > > > > > > > here. > > > > > > > > > > > > > > > > > > > > NaNs are less common in the SQL world than in the DataFrame > > > > world > > > > > > > where > > > > > > > > > > NaNs were used for a long time to represent missing values. > > > > They > > > > > > > still > > > > > > > > > > exist with different canonical representations and > > different > > > > sign > > > > > > > > bits. I > > > > > > > > > > agree it might not be correct semantically, but sadly that > > is > > > > the > > > > > > > world > > > > > > > > > we > > > > > > > > > > deal with. NumPy and Numba do not have missing data > > > > > functionality, > > > > > > > > people > > > > > > > > > > use NaNs there, and people definitely use that in their > > > > > analytical > > > > > > > > > > dataflows. Another point that was brought up in the GH > > > > discussion > > > > > > was > > > > > > > > > "what > > > > > > > > > > about infinity? You could argue that having infinity in > > > > > statistics > > > > > > is > > > > > > > > > > similarly unuseful as it's too wide of a bound". I would > > > argue > > > > > that > > > > > > > > > > infinity is very different as there is no discussion on > > what > > > > the > > > > > > > > ordering > > > > > > > > > > or pattern of infinity is. Everyone agrees that `min(1.0, > > > inf, > > > > > > -inf) > > > > > > > == > > > > > > > > > > -inf` and each infinity only has a single bit pattern. > > > > > > > > > > > > > > > > > > > > > It gives a defined order to every bit pattern and thus > > > > yields a > > > > > > > total > > > > > > > > > > order, mathematically speaking, which has value by itself. > > > With > > > > > NaN > > > > > > > > > counts, > > > > > > > > > > it was still undefined how different bit patterns of NaNs > > > were > > > > > > > supposed > > > > > > > > > to > > > > > > > > > > be ordered, whether NaN was allowed to have a sign bit, > > etc., > > > > > > risking > > > > > > > > > that > > > > > > > > > > different engines could come to different results while > > > > filtering > > > > > > or > > > > > > > > > > sorting values within a file. > > > > > > > > > > > > > > > > > > > > Since the proposal phrases it as a goal to work "regardless > > > of > > > > > how > > > > > > > they > > > > > > > > > > order NaN w.r.t. other values" this statement feels > > > > out-of-place > > > > > to > > > > > > > me. > > > > > > > > > > Most hardware and most people don't care about total > > ordering > > > > and > > > > > > > > needing > > > > > > > > > > to take it into account while filtering using statistics > > > seems > > > > > like > > > > > > > > > > preferring the special case instead of the common case. > > > Almost > > > > > > noone > > > > > > > > > > filters for specific NaN value bit-patterns. SQL engines > > that > > > > > don't > > > > > > > > have > > > > > > > > > > IEEE total ordering as their default ordering for floats > > will > > > > > also > > > > > > > need > > > > > > > > > to > > > > > > > > > > do more special handling for this. > > > > > > > > > > > > > > > > > > > > I also agree with my colleague that doing an approach that > > is > > > > 50% > > > > > > of > > > > > > > > the > > > > > > > > > > way there will make the barrier to improving it to what it > > > > > actually > > > > > > > > > should > > > > > > > > > > be later on much higher. > > > > > > > > > > > > > > > > > > > > As for ways forward, I propose merging the `nan_count` and > > > > `sort > > > > > > > > > ordering` > > > > > > > > > > proposals into one to make one proposal, as they are linked > > > > > > together, > > > > > > > > and > > > > > > > > > > moving forward with one without knowing what will happen to > > > the > > > > > > other > > > > > > > > > seems > > > > > > > > > > unwise. From a Polars perspective, having a `nan_count` and > > > > > > defining > > > > > > > > what > > > > > > > > > > happens to the `min` and `max` statistics when a page > > > contains > > > > > only > > > > > > > > NaNs > > > > > > > > > is > > > > > > > > > > enough to allow for all predicate filtering. I think, but > > > > correct > > > > > > me > > > > > > > > if I > > > > > > > > > > am wrong, this is also enough for all SQL engines that > > don't > > > > use > > > > > > > total > > > > > > > > > > ordering. But if you want to be impartial to the engine's > > > > > > > > floating-point > > > > > > > > > > ordering and allow engines with total ordering to do > > > inequality > > > > > > > filters > > > > > > > > > > when `nan_count > 0` you would need a `positive_nan_count` > > > and > > > > a > > > > > > > > > > `negative_nan_count`. I understand the downside with Thrift > > > > > > > complexity, > > > > > > > > > but > > > > > > > > > > introducing another sort order is also adding complexity > > just > > > > in > > > > > a > > > > > > > > > > different place. > > > > > > > > > > > > > > > > > > > > I would really like to see this move forward, so I hope > > these > > > > > > > concerns > > > > > > > > > help > > > > > > > > > > move it forward towards a solution that works for everyone. > > > > > > > > > > > > > > > > > > > > Kind regards, > > > > > > > > > > Gijs > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 31, 2025 at 7:46 PM Andrew Lamb < > > > > > > [email protected]> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > I would also be in favor of starting a vote > > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 31, 2025 at 11:23 AM Jan Finis < > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > As the author of both the IEEE754 total order > > > > > > > > > > > > <https://github.com/apache/parquet-format/pull/221> PR > > > and > > > > > the > > > > > > > > > earlier > > > > > > > > > > > PR > > > > > > > > > > > > that basically proposed `nan_count` > > > > > > > > > > > > <https://github.com/apache/parquet-format/pull/196>, > > my > > > > > > current > > > > > > > > vote > > > > > > > > > > > would > > > > > > > > > > > > be for IEEE754 total order. > > > > > > > > > > > > Consequently, I would like to request a formal vote for > > > the > > > > > PR > > > > > > > > > > > introducing > > > > > > > > > > > > IEEE754 total order ( > > > > > > > > > https://github.com/apache/parquet-format/pull/221 > > > > > > > > > > ), > > > > > > > > > > > > if > > > > > > > > > > > > that is possible. > > > > > > > > > > > > > > > > > > > > > > > > My Rationales: > > > > > > > > > > > > > > > > > > > > > > > > - It's conceptually simpler. It's easier to explain. > > > > It's > > > > > > > based > > > > > > > > on > > > > > > > > > > an > > > > > > > > > > > > IEEE-standardized order predicate. > > > > > > > > > > > > - There are already multiple implementations showing > > > > > > > > feasibility. > > > > > > > > > > This > > > > > > > > > > > > will likely make the adoption quicker. > > > > > > > > > > > > - It gives a defined order to every bit pattern and > > > thus > > > > > > > yields > > > > > > > > a > > > > > > > > > > > total > > > > > > > > > > > > order, mathematically speaking, which has value by > > > > itself. > > > > > > > With > > > > > > > > > NaN > > > > > > > > > > > > counts, > > > > > > > > > > > > it was still undefined how different bit patterns of > > > > NaNs > > > > > > were > > > > > > > > > > > supposed > > > > > > > > > > > > to > > > > > > > > > > > > be ordered, whether NaN was allowed to have a sign > > > bit, > > > > > > etc., > > > > > > > > > > risking > > > > > > > > > > > > that > > > > > > > > > > > > different engines could come to different results > > > while > > > > > > > > filtering > > > > > > > > > or > > > > > > > > > > > > sorting values within a file. > > > > > > > > > > > > - It also solves sort order completely. With > > > nan_counts > > > > > > only, > > > > > > > it > > > > > > > > > is > > > > > > > > > > > > still undefined whether nans should be sorted before > > > or > > > > > > after > > > > > > > > all > > > > > > > > > > > values > > > > > > > > > > > > (or both, depending on sign bit), so any file > > > including > > > > > NaNs > > > > > > > > could > > > > > > > > > > not > > > > > > > > > > > > really leverage sort order without being ambiguous. > > > > > > > > > > > > - It's less complex in thrift. Having fields that > > only > > > > > apply > > > > > > > to > > > > > > > > a > > > > > > > > > > > > handful of data types is somehow weird. If every > > type > > > > did > > > > > > > this, > > > > > > > > we > > > > > > > > > > > would > > > > > > > > > > > > have a plethora of non-generic fields in thrift. > > > > > > > > > > > > - Treating NaNs so specially is giving them > > attention > > > > they > > > > > > > don't > > > > > > > > > > > > deserve. Most data sets do not contain NaNs. If a > > use > > > > case > > > > > > > > really > > > > > > > > > > > > requires > > > > > > > > > > > > them and needs filtering to ignore them, they can > > > store > > > > > NULL > > > > > > > > > > instead, > > > > > > > > > > > > or encode them differently. I would prefer the > > average > > > > > case > > > > > > > over > > > > > > > > > the > > > > > > > > > > > > special case here. > > > > > > > > > > > > - The majority of the people discussing this so far > > > seem > > > > > to > > > > > > > > favor > > > > > > > > > > > total > > > > > > > > > > > > order. > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > Jan > > > > > > > > > > > > > > > > > > > > > > > > Am Sa., 26. Juli 2025 um 17:38 Uhr schrieb Gang Wu < > > > > > > > > [email protected] > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > > > As this discussion has been open for more than two > > > years, > > > > > I’d > > > > > > > > like > > > > > > > > > to > > > > > > > > > > > > bump > > > > > > > > > > > > > up > > > > > > > > > > > > > this thread again to update the progress and collect > > > > > > feedback. > > > > > > > > > > > > > > > > > > > > > > > > > > *Background* > > > > > > > > > > > > > • Today Parquet’s min/max stats and page index omit > > > NaNs > > > > > > > > entirely. > > > > > > > > > > > > > • Engines can’t safely prune floating values because > > > they > > > > > > know > > > > > > > > > > nothing > > > > > > > > > > > on > > > > > > > > > > > > > NaNs. > > > > > > > > > > > > > • Column index is disabled if any page contains only > > > > NaNs. > > > > > > > > > > > > > > > > > > > > > > > > > > There are two active proposals as below: > > > > > > > > > > > > > > > > > > > > > > > > > > *Proposal A - IEEE754TotalOrder* (from the PR [1]) > > > > > > > > > > > > > • Define a new ColumnOrder to include +0, –0 and all > > > NaN > > > > > > > > > > bit‐patterns. > > > > > > > > > > > > > • Stats and column index store NaNs if they appear. > > > > > > > > > > > > > • Three PoC impls are ready: arrow-rs [2], duckdb [3] > > > and > > > > > > > > > > parquet-java > > > > > > > > > > > > [4]. > > > > > > > > > > > > > • For more context of this approach, please refer to > > > > > > discussion > > > > > > > > in > > > > > > > > > > [5]. > > > > > > > > > > > > > > > > > > > > > > > > > > *Proposal B - add nan_count* (from a comment [6] to > > > [1]) > > > > > > > > > > > > > • Add `nan_count` to stats and a `nan_counts` list to > > > > > column > > > > > > > > index. > > > > > > > > > > > > > • For all‐NaNs cases, write NaN to min/max and use > > > > > nan_count > > > > > > to > > > > > > > > > > > > > distinguish. > > > > > > > > > > > > > > > > > > > > > > > > > > Both solutions have pros and cons but are way better > > > than > > > > > the > > > > > > > > > status > > > > > > > > > > > quo > > > > > > > > > > > > > today. > > > > > > > > > > > > > Please share your thoughts on the two proposals > > above, > > > or > > > > > > maybe > > > > > > > > > come > > > > > > > > > > up > > > > > > > > > > > > > with > > > > > > > > > > > > > better alternatives. We need consensus on one > > proposal > > > > and > > > > > > move > > > > > > > > > > > forward. > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > https://github.com/apache/parquet-format/pull/221 > > > > > > > > > > > > > [2] https://github.com/apache/arrow-rs/pull/7408 > > > > > > > > > > > > > [3] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/duckdb/duckdb/compare/main...Mytherin:duckdb:ieeeorder > > > > > > > > > > > > > [4] https://github.com/apache/parquet-java/pull/3191 > > > > > > > > > > > > > [5] > > https://github.com/apache/parquet-format/pull/196 > > > > > > > > > > > > > [6] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/parquet-format/pull/221#issuecomment-2931376077 > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > Gang > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 28, 2023 at 4:22 PM Jan Finis < > > > > > [email protected] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Dear contributors, > > > > > > > > > > > > > > > > > > > > > > > > > > > > My PR has now gathered comments for a week and the > > > gist > > > > > of > > > > > > > all > > > > > > > > > open > > > > > > > > > > > > > issues > > > > > > > > > > > > > > is the question of how to encode pages/column > > chunks > > > > that > > > > > > > > contain > > > > > > > > > > > only > > > > > > > > > > > > > > NaNs. There are different suggestions and I don't > > see > > > > one > > > > > > > > common > > > > > > > > > > > > favorite > > > > > > > > > > > > > > yet. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I have outlined three alternatives of how we can > > > handle > > > > > > these > > > > > > > > > and I > > > > > > > > > > > > want > > > > > > > > > > > > > us > > > > > > > > > > > > > > to reach a conclusion here, so I can update my PR > > > > > > accordingly > > > > > > > > and > > > > > > > > > > > move > > > > > > > > > > > > on > > > > > > > > > > > > > > with it. As this is my first contribution to > > > parquet, I > > > > > > don't > > > > > > > > > know > > > > > > > > > > > the > > > > > > > > > > > > > > decision processes here. Do we vote? Is there a > > > single > > > > or > > > > > > > group > > > > > > > > > of > > > > > > > > > > > > > decision > > > > > > > > > > > > > > makers? *Please let me know how to come to a > > > conclusion > > > > > > here; > > > > > > > > > what > > > > > > > > > > > are > > > > > > > > > > > > > the > > > > > > > > > > > > > > next steps?* > > > > > > > > > > > > > > > > > > > > > > > > > > > > For reference, here are the three alternatives I > > > > pointed > > > > > > out. > > > > > > > > You > > > > > > > > > > can > > > > > > > > > > > > > find > > > > > > > > > > > > > > detailed description of their PROs and CONs in my > > > > > comment: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/parquet-format/pull/196#issuecomment-1486416762 > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. My initial proposal, i.e., encoding only-NaN > > pages > > > > by > > > > > > > > > > min=max=NaN. > > > > > > > > > > > > > > 2. Adding `num_values` to the ColumnIndex, to make > > it > > > > > > > symmetric > > > > > > > > > > with > > > > > > > > > > > > > > Statistics in pages & `ColumnMetaData` and to > > enable > > > > the > > > > > > > > > > computation > > > > > > > > > > > > > > `num_values - null_count - nan_count == 0` > > > > > > > > > > > > > > 3. Adding a `nan_pages` bool list to the column > > > index, > > > > > > which > > > > > > > > > > > indicates > > > > > > > > > > > > > > whether a page contains only NaNs > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers > > > > > > > > > > > > > > Jan Finis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
