What do you consider tiny for a parquet file? Few KB or MB? 16 GB may seem like a lot but it is quite small for a query engine. If possible, increase the container to 32 GB and try giving it as much Heap as possible, since that's what it will use for the REST API. Try > 16 GB Heap and see if you still have issues.
Drill 1.18 is also starting to support Docker/K8s a little better so you might be able to setup parallel workers (drillbits) with ZooKeeper and spread the memory usage that way. Though that could take a bit of setting up. I'd try to increase the resources on your single drillbit before parallelizing it. - Rafael On Sun, Jun 28, 2020 at 6:17 PM Avner Levy <[email protected]> wrote: > > Hi, > I'm using Drill 1.18 (master) docker and trying to configure its memory > after getting out of heap memory errors: > "RESOURCE ERROR: There is not enough heap memory to run this query using > the web interface." > The docker is serving remote clients through the REST API. > The queries are simple selects over tiny parquet files that are stored in > S3. > It is running on in 16GB container, configured with a heap of 8GB, and 8GB > direct memory. > I tried to use: > exec.queue.enable=true > exec.queue.large=1 > exec.queue.small=1 > > and verified it was configured correctly, but I still see queries running > concurrently. > In addition, the "drill.queries.enqueued" counter remains zero. > Is this mechanism supported in drill-embedded? > > In addition, it seems there is some memory leak, since after a while even > with no query running for a while, running a single tiny query still gives > the same error. > Any insight would be highly appreciated :) > Thanks, > Avner
