Hi Giovanni,
The error given by the External-Sort indicates a problem while
spilling the excess memory into disk.
When you enlarged the memory (from the default 2GB) to 8GB the LAG query
may succeeded without spilling, hence circumvented the issue.
Yes, you can keep enlarging the memory and run w/o spilling, but better
check and fix the root issue.
How is your spilling configured - check the filesystem -
"drill.exec.spill.fs" and the directories - "drill.exec.spill.directories"
The default is the local filesystem, and into /tmp . It is possible
that very little disk space is available for /tmp .
Thanks,
-- Boaz
On 3/12/19 6:03 AM, Giovanni Conte wrote:
Hello,
I am doing a LAG and a SUM query over a PCAP dataset of 300Mb.
I get this error:
RESOURCE ERROR: External Sort encountered an error while spilling to disk
Then I changetd
planner.memory.max_query_memory_per_node ---> 8589934592
and with this I am able to perform the LAG but not the SUM.
I have no problem of memory since I am working on a server with 72 cores
and 256 GB of RAM.
Which other parameter should I change to avoid the RESOURCE ERROR?
Can I go over 8589934592?
Thank you very much,
Giovanni
