mhatipog commented on issue #27455:
URL: https://github.com/apache/datafusion/issues/27455#issuecomment-6921404161

   The same happens with float partition columns, where several valid spellings 
of one value are common: `-0` vs `-0.0` vs `0` vs `0.0`, `1e2` vs `100.0`, and 
`NaN` vs `nan`. In addition, `-Infinity` never matches, because the literal's 
`to_string()` is `-inf`. DataFusion 55.0.0:
   
   ```python
   import os, tempfile, pyarrow as pa, pyarrow.parquet as pq, datafusion
   root = tempfile.mkdtemp()
   for i, s in enumerate(["-0", "0.0", "1e2", "100.0", "-Infinity", "NaN"]):
       os.makedirs(f"{root}/d={s}"); pq.write_table(pa.table({"id": 
pa.array([i], pa.int64())}), f"{root}/d={s}/p.parquet")
   ctx = datafusion.SessionContext()
   ctx.register_parquet("t", root, table_partition_cols=[("d", pa.float64())], 
file_extension=".parquet")
   ctx.sql("SELECT id, d FROM t ORDER BY id").to_pylist()
   # d parsed as -0.0, 0.0, 100.0, 100.0, -inf, nan
   ctx.sql("SELECT id FROM t WHERE d = 0.0").to_pylist()                        
  # [1]      expected [0, 1]
   ctx.sql("SELECT id FROM t WHERE d = 100.0").to_pylist()                      
  # [3]      expected [2, 3]
   ctx.sql("SELECT id FROM t WHERE d = CAST('-Infinity' AS 
DOUBLE)").to_pylist()  # []       expected [4]
   ctx.sql("SELECT id FROM t WHERE d >= 0.0 AND d <= 0.0").to_pylist()          
  # [1]      (simplified to d = 0.0)
   ctx.sql("SELECT id FROM t WHERE d IN (0.0, 100.0)").to_pylist()              
  # [0, 1, 2, 3]  correct
   ```
   
   In a wider run on Linux with 16 spellings, `d = CAST('NaN' AS DOUBLE)` and 
`d = CAST('Infinity' AS DOUBLE)` (which miss the `nan` and `inf` directories) 
and `d BETWEEN -0.0 AND -0.0` fail the same way. DuckDB, Polars, 
pyarrow.dataset and Spark return the full set for every predicate there. So a 
fix that compares parsed values rather than strings (or that falls back to 
listing when the literal has more than one spelling) would also need to cover 
floats.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to