nuno-faria commented on issue #17597:
URL: https://github.com/apache/datafusion/issues/17597#issuecomment-3299735649
> What about:
>
> select l_comment
> from lineitem
> order by l_comment
> limit 50000;
It also runs out of memory:
```sql
> CREATE EXTERNAL TABLE lineitem STORED AS PARQUET LOCATION
'data/lineitem.parquet';
0 row(s) fetched.
Elapsed 0.004 seconds.
> select l_comment
from lineitem
order by l_comment
limit 50000;
memory allocation of 10018118656 bytes failed
```
> Even if the data is _perfectly_ sorted you still need to keep 50000 *
size_per_row in memory.
I found that when the data is already sorted on disk the issue does not
happen:
```sql
> select *
from lineitem
order by l_orderkey
limit 50000;
...
50000 row(s) fetched. (First 40 displayed. Use --maxrows to adjust)
Elapsed 0.348 seconds.
```
My reason for thinking its a bug is that the source file is only 200MB
compressed (snappy), while in that `select l_comment from lineitem order by
l_comment limit 50000` query above it allocates around 64GB before crashing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
