Hello Gang and others, I am willing to implement the C++ POC.
> 2026年3月14日 23:56，Gang Wu <[email protected]> 写道： > > Update: > > Java POC is ready for IEEE 754 column order combined with nan_count: > https://github.com/apache/parquet-java/pull/3393 > > The spec PR has been updated earlier to address all comments: > https://github.com/apache/parquet-format/pull/514 > > Really appreciate any review and feedback! > > Best, > Gang > > > > > On Wed, Feb 11, 2026 at 4:24 PM Gang Wu <[email protected]> wrote: > >> Hello all, >> >> I'm reaching out to help drive this long-running discussion—nearly >> three years now—towards a final resolution. With Jan's authorization, >> and my sincere thanks for his sustained effort, I want to help push >> this issue to the finish line. >> >> To recap, we have two primary proposals on how to handle NaNs in >> statistics and column indexes: >> >> * IEEE 754 Total Order [1]: Proposes adding a new column order >> IEEE754TotalOrder for FLOAT/DOUBLE/FLOAT16. This provides a defined >> ordering for every float bit pattern, including NaNs and -0/+0, >> allowing writers to include NaNs in min/max and removing ambiguity for >> only-NaN pages. >> * Combined Approach [2]: Proposes adopting the IEEE 754 total order >> alongside explicit nan_count(s) fields. This approach mandates the >> nan_count(s) when the new order is used and clarifies how to handle >> edge cases from legacy writers. >> >> Based on the recent comments, it appears the combined approach [2] is >> gaining consensus, although the IEEE 754 total order [1] still has >> strong advocates. >> >> I agree with the sentiment that technical direction should be made by >> consensus, not a vote. To that end, I'd like to solicit further >> feedback specifically on the combined approach [2] to see if we can >> achieve the necessary consensus to move forward now. >> >> I recall that the total order proposal [1] already has three PoC >> implementations. For the combined approach [2], I can draft a PoC in >> parquet-java, but to meet the two-implementation requirement, we would >> need one more contributor to step up. >> >> [1] https://github.com/apache/parquet-format/pull/221 >> [2] https://github.com/apache/parquet-format/pull/514 >> >> Best, >> Gang >> >> >> On Sat, Aug 16, 2025 at 1:59 AM Gijs Burghoorn <[email protected]> >> wrote: >>> >>> Hello Jan, >>> >>> Thank you for pushing this through. Apart from some smaller nits, we also >>> really like the current proposal. >>> >>> Thanks, >>> Gijs >>> >>> On Fri, Aug 15, 2025 at 3:33 PM Andrew Lamb <[email protected]> >> wrote: >>> >>>> I have started organizing a project[1] in arrow-rs 's Parquet reader >> to try >>>> and implement this proposal. >>>> >>>> Hopefully that can be 1 / 2 open source implementations needed. >>>> >>>> Thanks again for helping drive this along, >>>> Andrew >>>> >>>> [1] https://github.com/apache/arrow-rs/issues/8156 >>>> >>>> On Wed, Aug 13, 2025 at 5:39 AM Jan Finis <[email protected]> wrote: >>>> >>>>> I have now tagged >>>>> < >>>> >> https://github.com/apache/parquet-format/pull/514#issuecomment-3182978173 >>>>>> >>>>> the people that argued for total order in the initial PR. Let's see >> their >>>>> response. >>>>> >>>>> If I understand the adoption process correctly, the next hurdle to >>>> getting >>>>> this adopted is two open >>>>> source (!) implementations proving its feasibility. We already had >> that >>>> for >>>>> IEEE total order. If we >>>>> prefer the solution with nan counts, we'll need it there as well. I >>>> myself >>>>> work on a proprietary >>>>> implementation, so I'm counting on others here :). Be prepared >> though, >>>> this >>>>> will likely take months >>>>> unless the interest in this topic has risen to a point where people >> are >>>>> eager to jump on the implementation >>>>> right away. >>>>> >>>>> So, I guess it will take some months of soaking time before any >> formal >>>> vote >>>>> can be done >>>>> (given that we reach consensus that this is what we want and we find >>>> people >>>>> for the implementations). >>>>> >>>>> Cheers, >>>>> Jan >>>>> >>>>> Am Mi., 13. Aug. 2025 um 01:18 Uhr schrieb Ryan Blue < >> [email protected]>: >>>>> >>>>>> Thanks, Jan. I also went through the combined proposal and it looks >>>>> mostly >>>>>> good to me. >>>>>> >>>>>>> First of all, to make it quick: Yes, the solution of having >>>> nan_counts >>>>>> *and* total order, which was brought up multiple times, does work >> and >>>>>> solves more cases than just either of both. >>>>>> >>>>>> Great, then we have a solution for both filtering use cases and for >>>>> moving >>>>>> ahead with total order. And thanks to Andrew for suggesting this as >>>> well >>>>> on >>>>>> the second PR. I think this also looks like this is something that >>>> Orson >>>>> is >>>>>> okay with given his comments on the latest PR. >>>>>> >>>>>> Is there anyone against the combined approach? I don't see a big >>>> downside >>>>>> for anyone. It is compatible with previous stats rules, has a NaN >>>> count, >>>>>> and allows using either type-specific order or total order. >>>>>> >>>>>> Assuming that this satisfies the big objections, I think we should >> wait >>>>> for >>>>>> a few days to make sure everyone has time to check out the new PR >> and >>>>> then >>>>>> vote to adopt it. >>>>>> >>>>>> Ryan >>>>>> >>>>>> On Mon, Aug 11, 2025 at 6:03 AM Andrew Lamb < >> [email protected]> >>>>>> wrote: >>>>>> >>>>>>> Thank you Jan -- I read through the new combined proposal, and I >>>>> thought >>>>>> it >>>>>>> looks good and addresses the feedback so far. I left some small >> style >>>>>>> suggestions, but nothing that is required from my perspective >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Sat, Aug 9, 2025 at 9:07 AM Jan Finis <[email protected]> >> wrote: >>>>>>> >>>>>>>> Hey Ryan, >>>>>>>> >>>>>>>> Thanks for chiming in. First of all, to make it quick: Yes, the >>>>>> solution >>>>>>> of >>>>>>>> having nan_counts *and* total order, which was brought up >> multiple >>>>>> times, >>>>>>>> does work and solves more cases than just either of both. >>>>>>>> >>>>>>>> I strongly prefer continuing to discuss the merits of these >>>>> approaches >>>>>>>>> rather than trying to decide with a vote. >>>>>>>> >>>>>>>> >>>>>>>> In theory, I agree that it isn't good to silence a discussion >> by >>>> just >>>>>>>> voting for one possible solution and technical issues should be >>>>>>> discussed. >>>>>>>> However, please note that we have been circling on this for >> over >>>> two >>>>>>> years >>>>>>>> now, including an extended discussion that brought up all >> arguments >>>>>>>> multiple times. This is in stark contrast to the >>>>>>>> speed with which you guys work on the Iceberg spec, for >> example. >>>>> There, >>>>>>> you >>>>>>>> also do not discuss the merits of various solutions for >> multiple >>>>> years. >>>>>>> You >>>>>>>> just pick one and merge it after a *reasonable* time of >> discussion. >>>>>>>> If you had the speed we currently have here, nothing would get >>>> done. >>>>>>> Thus, >>>>>>>> I see this as a clear case of *"the perfect is the enemy of the >>>>> good"*. >>>>>>>> Yes, we can continue looking for the perfect solution, >>>>>>>> but that will likely lead to keeping us at the status quo, >> which is >>>>> the >>>>>>>> worst of them all. >>>>>>>> >>>>>>>> That being said, I'm also happy to create a PR which does both >>>> total >>>>>>> order >>>>>>>> and NaN counts; after all, I just want the issue solved and all >>>> these >>>>>>>> solutions are better than the status quo. >>>>>>>> >>>>>>>> *As this was now suggest by at least three people, I guess it's >>>> worth >>>>>>>> doing, so here you go: >>>>>> https://github.com/apache/parquet-format/pull/514 >>>>>>>> <https://github.com/apache/parquet-format/pull/514>* >>>>>>>> >>>>>>>> With this, we should have PRs covering most of the solution >> space. >>>>>>>> (I'm refusing to create a PR with negative and positive >> nan_counts; >>>>>>>> nan_counts + total order has to suffice; the complexity >> madness has >>>>> to >>>>>>> stop >>>>>>>> somewhere) >>>>>>>> I still believe that there was an amount of people who already >>>> found >>>>>>>> nan_counts too complex and therefore wanted IEEE total order, >> and >>>>> these >>>>>>>> people may not like putting on extra complexity, >>>>>>>> but let's see, maybe some have also changed their opinion in >> the >>>>>>> meantime. >>>>>>>> >>>>>>>> >>>>>>>> *Given all this, we can also first do an informal vote where >>>> everyone >>>>>> can >>>>>>>> vote for which of the three their favorite would be.Maybe a >> clear >>>>>>> favorite >>>>>>>> will emerge and then we can vote on this one.* >>>>>>>> >>>>>>>> But of course, we can also take some weeks to discuss the three >>>>>>> solutions, >>>>>>>> now that we have PRs for all of them. I just hope this won't >> make >>>> us >>>>>>>> continue for another 2 years, or an >>>>>>>> infinite stalemate where each solution is vetoed by a PMC >> member. >>>>>>>> (Sorry for becoming a bit cynical here; I have just spent way >> too >>>>> much >>>>>>> time >>>>>>>> of my life with double statistics at this point ;) ...) >>>>>>>> >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Jan >>>>>>>> >>>>>>>> Am Fr., 8. Aug. 2025 um 23:38 Uhr schrieb Ryan Blue < >>>>> [email protected] >>>>>>> : >>>>>>>> >>>>>>>>> Regarding the process for this, I strongly prefer continuing >> to >>>>>> discuss >>>>>>>> the >>>>>>>>> merits of these approaches rather than trying to decide with >> a >>>>> vote. >>>>>> I >>>>>>>>> don't think it is a good practice to use a vote to decide on >> a >>>>>>> technical >>>>>>>>> direction. There are very few situations that warrant it and >> I >>>>> don't >>>>>>>> think >>>>>>>>> that this is one of them. While this issue has been open for >> a >>>> long >>>>>>> time, >>>>>>>>> that appears to be the result of it not being anyone's top >>>> priority >>>>>>>> rather >>>>>>>>> than indecision. >>>>>>>>> >>>>>>>>> For the technical merits of these approaches, I think that >> we can >>>>>> find >>>>>>> a >>>>>>>>> middle ground. I agree with Jan that when working with sorted >>>>> values, >>>>>>> we >>>>>>>>> need to know how NaN values were handled and that requires >> using >>>> a >>>>>>>>> well-defined order that includes NaN and its variations >> (because >>>> we >>>>>>>> should >>>>>>>>> not normalize). Using NaN count is not sufficient for >> ordering >>>>> rows. >>>>>>>>> >>>>>>>>> Gijs also brings up good points about how NaN values show up >> in >>>>>> actual >>>>>>>>> datasets: not just when used in place of null, but also as >> the >>>>> result >>>>>>> of >>>>>>>>> normal calculations on abnormal data, like `sqrt(-4.0)` or >>>>>> `log(-1.0)`. >>>>>>>>> Both of those present problems when mixed with valid data >> because >>>>> of >>>>>>> the >>>>>>>>> stats "poisoning" problem, where the range of valid data is >>>> usable >>>>>>> until >>>>>>>> a >>>>>>>>> single NaN is mixed in. >>>>>>>>> >>>>>>>>> Another issue is that NaN is error-prone because "regular" >>>>> comparison >>>>>>> is >>>>>>>>> always false: >>>>>>>>> ``` >>>>>>>>> Math.log(-1.0) >= 2 => FALSE >>>>>>>>> Math.log(-1.0) < 2 => FALSE >>>>>>>>> 2 > Math.log(-1.0) => FALSE >>>>>>>>> ``` >>>>>>>>> >>>>>>>>> As a result, Iceberg doesn't trust NaN values as either >> lower or >>>>>> upper >>>>>>>>> bounds because we don't want to go back to the code that >> produced >>>>> the >>>>>>>> value >>>>>>>>> to see what the comparison order was to determine whether NaN >>>>> values >>>>>> go >>>>>>>>> before or after others. >>>>>>>>> >>>>>>>>> Total order solves the second issue in theory, but regular >>>>> comparison >>>>>>> is >>>>>>>>> prevalent and not obvious to developers. And it also doesn't >> help >>>>>> when >>>>>>>> NaN >>>>>>>>> is used instead of null. So using total order is not >> sufficient >>>> for >>>>>>> data >>>>>>>>> skipping. >>>>>>>>> >>>>>>>>> I think the right compromise is to use `min`, `max`, and >>>>> `nan_count` >>>>>>> for >>>>>>>>> data skipping stats (where min and max cannot be NaN) and >> total >>>>>>> ordering >>>>>>>>> for sorting values. That satisfies the data skipping use >> cases >>>> and >>>>>> also >>>>>>>>> gives us an ordering of unaltered values that we can reason >>>> about. >>>>>>>>> >>>>>>>>> Does anyone think that doesn't work? >>>>>>>>> >>>>>>>>> Ryan >>>>>>>>> >>>>>>>>> On Fri, Aug 1, 2025 at 8:57 AM Gang Wu <[email protected]> >> wrote: >>>>>>>>> >>>>>>>>>> Thanks Jan for your endless effort on this! >>>>>>>>>> >>>>>>>>>> I'm in favor of simplicity and generalism. I think we have >>>>> already >>>>>>>>> debated >>>>>>>>>> a lot >>>>>>>>>> for `nan_count` in [1] and [2] is the reflection of those >>>>>>> discussions. >>>>>>>>>> Therefore >>>>>>>>>> I am inclined to start a vote for [2] unless there is a >>>>>> significantly >>>>>>>>>> better >>>>>>>>>> proposal. >>>>>>>>>> >>>>>>>>>> I would suggest everyone interested in this discussion to >>>> attend >>>>>> the >>>>>>>>>> scheduled >>>>>>>>>> sync on Aug 6th (detailed below) to spread the word to the >>>>> broader >>>>>>>>>> community. >>>>>>>>>> If we can get a consensus on [2], I can help start the >> vote and >>>>>> move >>>>>>>>>> forward. >>>>>>>>>> >>>>>>>>>> *Apache Parquet Community Sync Wednesday, August 6 · 10:00 >> – >>>>>> 11:00am >>>>>>> * >>>>>>>>>> *Time zone: America/Los_Angeles* >>>>>>>>>> *Google Meet joining info Video call link: >>>>>>>>>> https://meet.google.com/bhe-rvan-qjk >>>>>>>>>> <https://meet.google.com/bhe-rvan-qjk> * >>>>>>>>>> >>>>>>>>>> [1] https://github.com/apache/parquet-format/pull/196 >>>>>>>>>> [2] https://github.com/apache/parquet-format/pull/221 >>>>>>>>>> >>>>>>>>>> Best, >>>>>>>>>> Gang >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Aug 1, 2025 at 6:16 PM Jan Finis < >> [email protected]> >>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Gijs, >>>>>>>>>>> >>>>>>>>>>> Thank you for bringing up concrete points, I'm happy to >>>> discuss >>>>>>> them >>>>>>>> in >>>>>>>>>>> detail. >>>>>>>>>>> >>>>>>>>>>> NaNs are less common in the SQL world than in the >> DataFrame >>>>> world >>>>>>>> where >>>>>>>>>>>> NaNs were used for a long time to represent missing >> values. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> You could transcode between NULL to NaN before reading >> and >>>>>> writing >>>>>>> to >>>>>>>>>>> Parquet. You basically mention yourself that NaNs were >> used >>>> for >>>>>>>> missing >>>>>>>>>>> values, i.e., what is commonly a NULL, which wasn't >>>> available. >>>>>> So, >>>>>>>>>>> semantically, transcoding to NULL would even be the sane >>>> thing >>>>> to >>>>>>> do. >>>>>>>>>> Yes, >>>>>>>>>>> that will cost you some cycles, but should be a rather >>>>>> lightweight >>>>>>>>>>> operation in comparison to most other operations, so I >> would >>>>>> argue >>>>>>>> that >>>>>>>>>> it >>>>>>>>>>> won't totally ruin your performance. Similarly, why >> should >>>>>> Parquet >>>>>>>> play >>>>>>>>>>> along with a "hack" that was done in other frameworks >> due to >>>>>>>>> shortcomings >>>>>>>>>>> of those frameworks? So from a philosophical point of >> view, I >>>>>> think >>>>>>>>>>> supporting NaNs better is the wrong thing to do. Rather, >> we >>>>>> should >>>>>>>> be a >>>>>>>>>>> forcing function to align others to better behavior, so >>>>> appling a >>>>>>> bit >>>>>>>>> of >>>>>>>>>>> force might in the long run make people use NULLs also in >>>>>>> DataFrames. >>>>>>>>>>> >>>>>>>>>>> Of course, your argument also goes into the direction of >>>>>>> pragmatism: >>>>>>>>> If a >>>>>>>>>>> large part of the data science world uses NaNs to encode >>>>> missing >>>>>>>>> values, >>>>>>>>>>> then maybe Parquet should accept this de-facto standard >>>> rather >>>>>> than >>>>>>>>>>> fighting it. That is indeed a valid point. The weight of >> it >>>> is >>>>>>>>> debatable >>>>>>>>>>> and my personal conclusion is that it's still not worth >> it, >>>> as >>>>>> you >>>>>>>> can >>>>>>>>>>> transcode between NULLs and NaNs, but I do agree with its >>>>>> validity. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Since the proposal phrases it as a goal to work >> "regardless >>>> of >>>>>> how >>>>>>>> they >>>>>>>>>>>> order NaN w.r.t. other values" this statement feels >>>>>> out-of-place >>>>>>> to >>>>>>>>> me. >>>>>>>>>>>> Most hardware and most people don't care about total >>>> ordering >>>>>> and >>>>>>>>>> needing >>>>>>>>>>>> to take it into account while filtering using >> statistics >>>>> seems >>>>>>> like >>>>>>>>>>>> preferring the special case instead of the common case. >>>>> Almost >>>>>>>> noone >>>>>>>>>>>> filters for specific NaN value bit-patterns. SQL >> engines >>>> that >>>>>>> don't >>>>>>>>>> have >>>>>>>>>>>> IEEE total ordering as their default ordering for >> floats >>>> will >>>>>>> also >>>>>>>>> need >>>>>>>>>>> to >>>>>>>>>>>> do more special handling for this. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I disagree with the conclusion this statement draws. The >>>>> current >>>>>>>>>> behavior, >>>>>>>>>>> and nan_counts without total ordering, pose a real >> problem >>>>> here, >>>>>>> even >>>>>>>>> for >>>>>>>>>>> engines that don't care about bit patterns. I do agree >> that >>>>> most >>>>>>>>> database >>>>>>>>>>> engines, including the one I'm working on, do not care >> about >>>>> bit >>>>>>>>> patterns >>>>>>>>>>> and/or sign bits. However, how can our database engine >> know >>>>>> whether >>>>>>>> the >>>>>>>>>>> writer of a Parquet file saw it the same way? It can't. >>>>>> Therefore, >>>>>>> it >>>>>>>>>>> cannot know whether a writer, for example, ordered NaNs >>>> before >>>>> or >>>>>>>> after >>>>>>>>>> all >>>>>>>>>>> other numbers, or maybe ordered them by sign bit. So, if >> our >>>>>>> database >>>>>>>>>>> engine now sees a float column in sorting columns, it >> cannot >>>>>> apply >>>>>>>> any >>>>>>>>>>> optimization without a lot of special casing, as it >> doesn't >>>>> know >>>>>>>>> whether >>>>>>>>>>> NaNs will be before all other values, after all other >> values, >>>>> or >>>>>>>> maybe >>>>>>>>>>> both, depending on sign bit. It could apply contrived >> logic >>>>> that >>>>>>>> tries >>>>>>>>> to >>>>>>>>>>> infer where NaNs were placed from the NaN counts of the >> first >>>>> and >>>>>>>> last >>>>>>>>>>> page, but doing so will be a lot of ugly code that also >> feels >>>>> to >>>>>> be >>>>>>>> in >>>>>>>>>> the >>>>>>>>>>> wrong place. I.e., I don't want to need to load pages or >> the >>>>> page >>>>>>>>> index, >>>>>>>>>>> just to reason about a sort order. >>>>>>>>>>> >>>>>>>>>>> SQL engines that don't have >>>>>>>>>>>> IEEE total ordering as their default ordering for >> floats >>>> will >>>>>>> also >>>>>>>>> need >>>>>>>>>>> to >>>>>>>>>>>> do more special handling for this. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> This code, which I would indeed need to write for our >> engine, >>>>> is >>>>>>>>>> comparably >>>>>>>>>>> trivial. Simply choose the largest possible bit pattern >> as >>>>>>> comparison >>>>>>>>> for >>>>>>>>>>> upper bounds filtering for NaN, and the smallest >> possible bit >>>>>>> pattern >>>>>>>>> for >>>>>>>>>>> lower bounds. It's not more than a few lines of code that >>>> check >>>>>>>>> whether a >>>>>>>>>>> filter is NaN and then replace its value with the >>>>> highest/lowest >>>>>>> NaN >>>>>>>>> bit >>>>>>>>>>> pattern. It is similarly trivial to the special casing I >> need >>>>> to >>>>>> do >>>>>>>>> with >>>>>>>>>>> nan_counts, and it is way more trivial than the extra >> code I >>>>>> would >>>>>>>> need >>>>>>>>>> to >>>>>>>>>>> write for sorting columns, as depicted above. >>>>>>>>>>> >>>>>>>>>>> From a Polars perspective, having a `nan_count` and >> defining >>>>> what >>>>>>>>>>>> happens to the `min` and `max` statistics when a page >>>>> contains >>>>>>> only >>>>>>>>>> NaNs >>>>>>>>>>> is >>>>>>>>>>>> enough to allow for all predicate filtering. I think, >> but >>>>>> correct >>>>>>>> me >>>>>>>>>> if I >>>>>>>>>>>> am wrong, this is also enough for all SQL engines that >>>> don't >>>>>> use >>>>>>>>> total >>>>>>>>>>>> ordering. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> It's not fully enough, as depicted above. Sorting columns >>>> would >>>>>>> still >>>>>>>>> not >>>>>>>>>>> work properly. >>>>>>>>>>> >>>>>>>>>>> As for ways forward, I propose merging the `nan_count` >> and >>>>> `sort >>>>>>>>>> ordering` >>>>>>>>>>>> proposals into one to make one proposal >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Note that the initial reason for proposing IEEE total >> order >>>> was >>>>>>> that >>>>>>>>>> people >>>>>>>>>>> in the discussion threads found nan_counts to be too >> complex >>>>> and >>>>>>> too >>>>>>>>> much >>>>>>>>>>> of an undeserving special case (re-read the discussion >> in the >>>>>>> initial >>>>>>>>> PR >>>>>>>>>>> <https://github.com/apache/parquet-format/pull/196> to >> see >>>> the >>>>>>>>>>> rationales). >>>>>>>>>>> So merging both together would go totally against the >> spirit >>>> of >>>>>> why >>>>>>>>> IEEE >>>>>>>>>>> total order was proposed. While it has further upsides, >> the >>>>> main >>>>>>>> reason >>>>>>>>>> was >>>>>>>>>>> indeed to *not have* nan_counts. If now the proposal >> would >>>> even >>>>>> go >>>>>>> to >>>>>>>>>>> positive and negative nan counts (i.e., even more >>>> complexity), >>>>>> this >>>>>>>>> would >>>>>>>>>>> go 180 degrees into the opposite direction of why people >>>> wanted >>>>>>> total >>>>>>>>>> order >>>>>>>>>>> in the first place. >>>>>>>>>>> >>>>>>>>>>> Cheers, >>>>>>>>>>> Jan >>>>>>>>>>> >>>>>>>>>>> Am Do., 31. Juli 2025 um 23:23 Uhr schrieb Gijs Burghoorn >>>>>>>>>>> <[email protected]>: >>>>>>>>>>> >>>>>>>>>>>> Hello Jan and others, >>>>>>>>>>>> >>>>>>>>>>>> First, let me preface by saying I am quite new here. >> So I >>>>>>> apologize >>>>>>>>> if >>>>>>>>>>>> there is some other better way to bring up these >> concerns. >>>> I >>>>>>>>> understand >>>>>>>>>>> it >>>>>>>>>>>> is very annoying to come in at the 11th hour and start >>>>> bringing >>>>>>> up >>>>>>>> a >>>>>>>>>>> bunch >>>>>>>>>>>> of concerns, but I would also like this to be done >> right. A >>>>>>>> colleague >>>>>>>>>> of >>>>>>>>>>>> mine brought up some concerns and alternative >> approaches in >>>>> the >>>>>>>>> GitHub >>>>>>>>>>>> thread; I will file some of the concerns here as a >>>> response. >>>>>>>>>>>> >>>>>>>>>>>>> Treating NaNs so specially is giving them attention >> they >>>>>> don't >>>>>>>>>> deserve. >>>>>>>>>>>> Most data sets do not contain NaNs. If a use case >> really >>>>>> requires >>>>>>>>> them >>>>>>>>>>> and >>>>>>>>>>>> needs filtering to ignore them, they can store NULL >>>> instead, >>>>> or >>>>>>>>> encode >>>>>>>>>>> them >>>>>>>>>>>> differently. I would prefer the average case over the >>>> special >>>>>>> case >>>>>>>>>> here. >>>>>>>>>>>> >>>>>>>>>>>> NaNs are less common in the SQL world than in the >> DataFrame >>>>>> world >>>>>>>>> where >>>>>>>>>>>> NaNs were used for a long time to represent missing >> values. >>>>>> They >>>>>>>>> still >>>>>>>>>>>> exist with different canonical representations and >>>> different >>>>>> sign >>>>>>>>>> bits. I >>>>>>>>>>>> agree it might not be correct semantically, but sadly >> that >>>> is >>>>>> the >>>>>>>>> world >>>>>>>>>>> we >>>>>>>>>>>> deal with. NumPy and Numba do not have missing data >>>>>>> functionality, >>>>>>>>>> people >>>>>>>>>>>> use NaNs there, and people definitely use that in their >>>>>>> analytical >>>>>>>>>>>> dataflows. Another point that was brought up in the GH >>>>>> discussion >>>>>>>> was >>>>>>>>>>> "what >>>>>>>>>>>> about infinity? You could argue that having infinity in >>>>>>> statistics >>>>>>>> is >>>>>>>>>>>> similarly unuseful as it's too wide of a bound". I >> would >>>>> argue >>>>>>> that >>>>>>>>>>>> infinity is very different as there is no discussion on >>>> what >>>>>> the >>>>>>>>>> ordering >>>>>>>>>>>> or pattern of infinity is. Everyone agrees that >> `min(1.0, >>>>> inf, >>>>>>>> -inf) >>>>>>>>> == >>>>>>>>>>>> -inf` and each infinity only has a single bit pattern. >>>>>>>>>>>> >>>>>>>>>>>>> It gives a defined order to every bit pattern and >> thus >>>>>> yields a >>>>>>>>> total >>>>>>>>>>>> order, mathematically speaking, which has value by >> itself. >>>>> With >>>>>>> NaN >>>>>>>>>>> counts, >>>>>>>>>>>> it was still undefined how different bit patterns of >> NaNs >>>>> were >>>>>>>>> supposed >>>>>>>>>>> to >>>>>>>>>>>> be ordered, whether NaN was allowed to have a sign bit, >>>> etc., >>>>>>>> risking >>>>>>>>>>> that >>>>>>>>>>>> different engines could come to different results while >>>>>> filtering >>>>>>>> or >>>>>>>>>>>> sorting values within a file. >>>>>>>>>>>> >>>>>>>>>>>> Since the proposal phrases it as a goal to work >> "regardless >>>>> of >>>>>>> how >>>>>>>>> they >>>>>>>>>>>> order NaN w.r.t. other values" this statement feels >>>>>> out-of-place >>>>>>> to >>>>>>>>> me. >>>>>>>>>>>> Most hardware and most people don't care about total >>>> ordering >>>>>> and >>>>>>>>>> needing >>>>>>>>>>>> to take it into account while filtering using >> statistics >>>>> seems >>>>>>> like >>>>>>>>>>>> preferring the special case instead of the common case. >>>>> Almost >>>>>>>> noone >>>>>>>>>>>> filters for specific NaN value bit-patterns. SQL >> engines >>>> that >>>>>>> don't >>>>>>>>>> have >>>>>>>>>>>> IEEE total ordering as their default ordering for >> floats >>>> will >>>>>>> also >>>>>>>>> need >>>>>>>>>>> to >>>>>>>>>>>> do more special handling for this. >>>>>>>>>>>> >>>>>>>>>>>> I also agree with my colleague that doing an approach >> that >>>> is >>>>>> 50% >>>>>>>> of >>>>>>>>>> the >>>>>>>>>>>> way there will make the barrier to improving it to >> what it >>>>>>> actually >>>>>>>>>>> should >>>>>>>>>>>> be later on much higher. >>>>>>>>>>>> >>>>>>>>>>>> As for ways forward, I propose merging the `nan_count` >> and >>>>>> `sort >>>>>>>>>>> ordering` >>>>>>>>>>>> proposals into one to make one proposal, as they are >> linked >>>>>>>> together, >>>>>>>>>> and >>>>>>>>>>>> moving forward with one without knowing what will >> happen to >>>>> the >>>>>>>> other >>>>>>>>>>> seems >>>>>>>>>>>> unwise. From a Polars perspective, having a >> `nan_count` and >>>>>>>> defining >>>>>>>>>> what >>>>>>>>>>>> happens to the `min` and `max` statistics when a page >>>>> contains >>>>>>> only >>>>>>>>>> NaNs >>>>>>>>>>> is >>>>>>>>>>>> enough to allow for all predicate filtering. I think, >> but >>>>>> correct >>>>>>>> me >>>>>>>>>> if I >>>>>>>>>>>> am wrong, this is also enough for all SQL engines that >>>> don't >>>>>> use >>>>>>>>> total >>>>>>>>>>>> ordering. But if you want to be impartial to the >> engine's >>>>>>>>>> floating-point >>>>>>>>>>>> ordering and allow engines with total ordering to do >>>>> inequality >>>>>>>>> filters >>>>>>>>>>>> when `nan_count > 0` you would need a >> `positive_nan_count` >>>>> and >>>>>> a >>>>>>>>>>>> `negative_nan_count`. I understand the downside with >> Thrift >>>>>>>>> complexity, >>>>>>>>>>> but >>>>>>>>>>>> introducing another sort order is also adding >> complexity >>>> just >>>>>> in >>>>>>> a >>>>>>>>>>>> different place. >>>>>>>>>>>> >>>>>>>>>>>> I would really like to see this move forward, so I hope >>>> these >>>>>>>>> concerns >>>>>>>>>>> help >>>>>>>>>>>> move it forward towards a solution that works for >> everyone. >>>>>>>>>>>> >>>>>>>>>>>> Kind regards, >>>>>>>>>>>> Gijs >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Jul 31, 2025 at 7:46 PM Andrew Lamb < >>>>>>>> [email protected]> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I would also be in favor of starting a vote >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Jul 31, 2025 at 11:23 AM Jan Finis < >>>>>> [email protected]> >>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> As the author of both the IEEE754 total order >>>>>>>>>>>>>> <https://github.com/apache/parquet-format/pull/221> >> PR >>>>> and >>>>>>> the >>>>>>>>>>> earlier >>>>>>>>>>>>> PR >>>>>>>>>>>>>> that basically proposed `nan_count` >>>>>>>>>>>>>> <https://github.com/apache/parquet-format/pull/196 >>> , >>>> my >>>>>>>> current >>>>>>>>>> vote >>>>>>>>>>>>> would >>>>>>>>>>>>>> be for IEEE754 total order. >>>>>>>>>>>>>> Consequently, I would like to request a formal >> vote for >>>>> the >>>>>>> PR >>>>>>>>>>>>> introducing >>>>>>>>>>>>>> IEEE754 total order ( >>>>>>>>>>> https://github.com/apache/parquet-format/pull/221 >>>>>>>>>>>> ), >>>>>>>>>>>>>> if >>>>>>>>>>>>>> that is possible. >>>>>>>>>>>>>> >>>>>>>>>>>>>> My Rationales: >>>>>>>>>>>>>> >>>>>>>>>>>>>> - It's conceptually simpler. It's easier to >> explain. >>>>>> It's >>>>>>>>> based >>>>>>>>>> on >>>>>>>>>>>> an >>>>>>>>>>>>>> IEEE-standardized order predicate. >>>>>>>>>>>>>> - There are already multiple implementations >> showing >>>>>>>>>> feasibility. >>>>>>>>>>>> This >>>>>>>>>>>>>> will likely make the adoption quicker. >>>>>>>>>>>>>> - It gives a defined order to every bit pattern >> and >>>>> thus >>>>>>>>> yields >>>>>>>>>> a >>>>>>>>>>>>> total >>>>>>>>>>>>>> order, mathematically speaking, which has value >> by >>>>>> itself. >>>>>>>>> With >>>>>>>>>>> NaN >>>>>>>>>>>>>> counts, >>>>>>>>>>>>>> it was still undefined how different bit >> patterns of >>>>>> NaNs >>>>>>>> were >>>>>>>>>>>>> supposed >>>>>>>>>>>>>> to >>>>>>>>>>>>>> be ordered, whether NaN was allowed to have a >> sign >>>>> bit, >>>>>>>> etc., >>>>>>>>>>>> risking >>>>>>>>>>>>>> that >>>>>>>>>>>>>> different engines could come to different >> results >>>>> while >>>>>>>>>> filtering >>>>>>>>>>> or >>>>>>>>>>>>>> sorting values within a file. >>>>>>>>>>>>>> - It also solves sort order completely. With >>>>> nan_counts >>>>>>>> only, >>>>>>>>> it >>>>>>>>>>> is >>>>>>>>>>>>>> still undefined whether nans should be sorted >> before >>>>> or >>>>>>>> after >>>>>>>>>> all >>>>>>>>>>>>> values >>>>>>>>>>>>>> (or both, depending on sign bit), so any file >>>>> including >>>>>>> NaNs >>>>>>>>>> could >>>>>>>>>>>> not >>>>>>>>>>>>>> really leverage sort order without being >> ambiguous. >>>>>>>>>>>>>> - It's less complex in thrift. Having fields >> that >>>> only >>>>>>> apply >>>>>>>>> to >>>>>>>>>> a >>>>>>>>>>>>>> handful of data types is somehow weird. If every >>>> type >>>>>> did >>>>>>>>> this, >>>>>>>>>> we >>>>>>>>>>>>> would >>>>>>>>>>>>>> have a plethora of non-generic fields in thrift. >>>>>>>>>>>>>> - Treating NaNs so specially is giving them >>>> attention >>>>>> they >>>>>>>>> don't >>>>>>>>>>>>>> deserve. Most data sets do not contain NaNs. If >> a >>>> use >>>>>> case >>>>>>>>>> really >>>>>>>>>>>>>> requires >>>>>>>>>>>>>> them and needs filtering to ignore them, they >> can >>>>> store >>>>>>> NULL >>>>>>>>>>>> instead, >>>>>>>>>>>>>> or encode them differently. I would prefer the >>>> average >>>>>>> case >>>>>>>>> over >>>>>>>>>>> the >>>>>>>>>>>>>> special case here. >>>>>>>>>>>>>> - The majority of the people discussing this so >> far >>>>> seem >>>>>>> to >>>>>>>>>> favor >>>>>>>>>>>>> total >>>>>>>>>>>>>> order. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>> Jan >>>>>>>>>>>>>> >>>>>>>>>>>>>> Am Sa., 26. Juli 2025 um 17:38 Uhr schrieb Gang Wu >> < >>>>>>>>>> [email protected] >>>>>>>>>>>> : >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> As this discussion has been open for more than >> two >>>>> years, >>>>>>> I’d >>>>>>>>>> like >>>>>>>>>>> to >>>>>>>>>>>>>> bump >>>>>>>>>>>>>>> up >>>>>>>>>>>>>>> this thread again to update the progress and >> collect >>>>>>>> feedback. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *Background* >>>>>>>>>>>>>>> • Today Parquet’s min/max stats and page index >> omit >>>>> NaNs >>>>>>>>>> entirely. >>>>>>>>>>>>>>> • Engines can’t safely prune floating values >> because >>>>> they >>>>>>>> know >>>>>>>>>>>> nothing >>>>>>>>>>>>> on >>>>>>>>>>>>>>> NaNs. >>>>>>>>>>>>>>> • Column index is disabled if any page contains >> only >>>>>> NaNs. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> There are two active proposals as below: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *Proposal A - IEEE754TotalOrder* (from the PR >> [1]) >>>>>>>>>>>>>>> • Define a new ColumnOrder to include +0, –0 and >> all >>>>> NaN >>>>>>>>>>>> bit‐patterns. >>>>>>>>>>>>>>> • Stats and column index store NaNs if they >> appear. >>>>>>>>>>>>>>> • Three PoC impls are ready: arrow-rs [2], >> duckdb [3] >>>>> and >>>>>>>>>>>> parquet-java >>>>>>>>>>>>>> [4]. >>>>>>>>>>>>>>> • For more context of this approach, please >> refer to >>>>>>>> discussion >>>>>>>>>> in >>>>>>>>>>>> [5]. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *Proposal B - add nan_count* (from a comment [6] >> to >>>>> [1]) >>>>>>>>>>>>>>> • Add `nan_count` to stats and a `nan_counts` >> list to >>>>>>> column >>>>>>>>>> index. >>>>>>>>>>>>>>> • For all‐NaNs cases, write NaN to min/max and >> use >>>>>>> nan_count >>>>>>>> to >>>>>>>>>>>>>>> distinguish. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Both solutions have pros and cons but are way >> better >>>>> than >>>>>>> the >>>>>>>>>>> status >>>>>>>>>>>>> quo >>>>>>>>>>>>>>> today. >>>>>>>>>>>>>>> Please share your thoughts on the two proposals >>>> above, >>>>> or >>>>>>>> maybe >>>>>>>>>>> come >>>>>>>>>>>> up >>>>>>>>>>>>>>> with >>>>>>>>>>>>>>> better alternatives. We need consensus on one >>>> proposal >>>>>> and >>>>>>>> move >>>>>>>>>>>>> forward. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1] >>>> https://github.com/apache/parquet-format/pull/221 >>>>>>>>>>>>>>> [2] https://github.com/apache/arrow-rs/pull/7408 >>>>>>>>>>>>>>> [3] >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >> https://github.com/duckdb/duckdb/compare/main...Mytherin:duckdb:ieeeorder >>>>>>>>>>>>>>> [4] >> https://github.com/apache/parquet-java/pull/3191 >>>>>>>>>>>>>>> [5] >>>> https://github.com/apache/parquet-format/pull/196 >>>>>>>>>>>>>>> [6] >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >> https://github.com/apache/parquet-format/pull/221#issuecomment-2931376077 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>> Gang >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Mar 28, 2023 at 4:22 PM Jan Finis < >>>>>>> [email protected] >>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Dear contributors, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> My PR has now gathered comments for a week and >> the >>>>> gist >>>>>>> of >>>>>>>>> all >>>>>>>>>>> open >>>>>>>>>>>>>>> issues >>>>>>>>>>>>>>>> is the question of how to encode pages/column >>>> chunks >>>>>> that >>>>>>>>>> contain >>>>>>>>>>>>> only >>>>>>>>>>>>>>>> NaNs. There are different suggestions and I >> don't >>>> see >>>>>> one >>>>>>>>>> common >>>>>>>>>>>>>> favorite >>>>>>>>>>>>>>>> yet. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I have outlined three alternatives of how we >> can >>>>> handle >>>>>>>> these >>>>>>>>>>> and I >>>>>>>>>>>>>> want >>>>>>>>>>>>>>> us >>>>>>>>>>>>>>>> to reach a conclusion here, so I can update my >> PR >>>>>>>> accordingly >>>>>>>>>> and >>>>>>>>>>>>> move >>>>>>>>>>>>>> on >>>>>>>>>>>>>>>> with it. As this is my first contribution to >>>>> parquet, I >>>>>>>> don't >>>>>>>>>>> know >>>>>>>>>>>>> the >>>>>>>>>>>>>>>> decision processes here. Do we vote? Is there a >>>>> single >>>>>> or >>>>>>>>> group >>>>>>>>>>> of >>>>>>>>>>>>>>> decision >>>>>>>>>>>>>>>> makers? *Please let me know how to come to a >>>>> conclusion >>>>>>>> here; >>>>>>>>>>> what >>>>>>>>>>>>> are >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> next steps?* >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> For reference, here are the three alternatives >> I >>>>>> pointed >>>>>>>> out. >>>>>>>>>> You >>>>>>>>>>>> can >>>>>>>>>>>>>>> find >>>>>>>>>>>>>>>> detailed description of their PROs and CONs in >> my >>>>>>> comment: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >> https://github.com/apache/parquet-format/pull/196#issuecomment-1486416762 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 1. My initial proposal, i.e., encoding only-NaN >>>> pages >>>>>> by >>>>>>>>>>>> min=max=NaN. >>>>>>>>>>>>>>>> 2. Adding `num_values` to the ColumnIndex, to >> make >>>> it >>>>>>>>> symmetric >>>>>>>>>>>> with >>>>>>>>>>>>>>>> Statistics in pages & `ColumnMetaData` and to >>>> enable >>>>>> the >>>>>>>>>>>> computation >>>>>>>>>>>>>>>> `num_values - null_count - nan_count == 0` >>>>>>>>>>>>>>>> 3. Adding a `nan_pages` bool list to the column >>>>> index, >>>>>>>> which >>>>>>>>>>>>> indicates >>>>>>>>>>>>>>>> whether a page contains only NaNs >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Cheers >>>>>>>>>>>>>>>> Jan Finis >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>
