And with logs as attachments.
On Sun, Jan 28, 2018 at 9:40 PM, Francis McGregor-Macdonald <
fran...@mc-mac.com> wrote:
> Thanks Paul and Kunal,
> I think I have the right information now. With Paul's changes (and fixing
> up a zoo.cfg error) it isn't crashing, rather failing. Logs attached, still
> blowing past memory limits. It does the same thing when re-running the
> query from the web console so presumably its not actually Tableau related
> despite me first generating it that way.
>
> Thanks.
>
> On Sat, Jan 27, 2018 at 1:15 PM, Francis McGregor-Macdonald <
> fran...@mc-mac.com> wrote:
>
>> Thanks Paul,
>>
>> I will update with your suggested memory allocations also and retry.
>>
>> Zookeeper crashed too which might explain more? I have attached the logs
>> from Zookeeper too.
>>
>> Thanks
>>
>> On Sat, Jan 27, 2018 at 6:45 AM, Paul Rogers <par0...@yahoo.com> wrote:
>>
>>> Hi Francis,
>>>
>>> Thanks much for the log. The log shows running a query, then immediately
>>> shows entries that occur when starting Drill. I'm guessing that Drill
>>> literally crashed at this point? This is more severe than the usual error
>>> in which a query exhausts memory.
>>>
>>> Some general observations. The Drill memory is 60 GB, but system memory
>>> is 61 GB. Perhaps try dropping total Drill memory some to give the OS and
>>> other tasks more headroom. For a SELECT * memory, Drill needs far less than
>>> what you have, so maybe try giving Drill 48 GB total.
>>>
>>> Then, Drill needs direct memory much more than heap. So, maybe give
>>> Drill 39 GB direct, 8 GB heap and 1 GB (the default) for code cache. These
>>> settings are in drill-env.sh.
>>>
>>> Kunal, you have more experience with these issues. Can you make
>>> additional suggestions by looking at the log?
>>>
>>> Thanks,
>>>
>>> - Paul
>>>
>>>
>>>
>>> On Thursday, January 25, 2018, 10:20:29 PM PST, Francis
>>> McGregor-Macdonald <fran...@mc-mac.com> wrote:
>>>
>>>
>>> Hi all,
>>>
>>> I am guessing that each of your EMR nodes are quite large? EMR nodes
>>> are: r4.2xlarge ('vcpu': 8, 'memory': 61)
>>>
>>> Property "planner.width.max_per_node" is set to = 6
>>>
>>> What is the system memory and what are the allocations for heap and
>>> direct?
>>> System Memory: 61GB (EMR nodes above)
>>> drill_mem_heap: 12G
>>> drill_mem_max: 48G
>>>
>>> The view is simple: SELECT * FROM s3://myparquet.parquet (14GB)
>>>
>>> planner.memory.max_query_memor y_per_node = 10479720202
>>>
>>> Drillbit.log attached (I think I have the correct selection included).
>>>
>>> Thanks
>>>
>>> On Fri, Jan 26, 2018 at 2:41 PM, Kunal Khatua <kkha...@mapr.com> wrote:
>>>
>>> What is the system memory and what are the allocations for heap and
>>> direct? The memory crash might be occurring due to insufficient heap. The
>>> limits parameter applies to the direct memory and not Heap.
>>>
>>> Can you share details in the logs from the crash?
>>>
>>> -----Original Message-----
>>> From: Timothy Farkas [mailto:tfar...@mapr.com]
>>> Sent: Thursday, January 25, 2018 2:58 PM
>>> To: user@drill.apache.org
>>> Subject: Re: Creating a Tableau extracts with Drill 1.12 uses unlimited
>>> memory
>>>
>>> Hi Francis,
>>>
>>> I am guessing that each of your EMR nodes are quite large (32 or 64
>>> vcpus). On large machines Drill's planner over parallelizes and over
>>> allocates memory. There is a property "planner.width.max_per_node" which
>>> limits the number of operators that can simultaneously execute on a
>>> Drillbit for a query. If you configure the width per node to something like
>>> 5 or 10 (you may have to play around with it) things should start working.
>>>
>>> Thanks,
>>> Tim
>>>
>>> ______________________________ __
>>> From: Francis McGregor-Macdonald <fran...@mc-mac.com>
>>> Sent: Thursday, January 25, 2018 1:58:22 PM
>>> To: user@drill.apache.org
>>> Subject: Creating a Tableau extracts with Drill 1.12 uses unlimited
>>> memory
>>>
>>> Creating a creating a Tableau (with 10.3, 10.5 desktop) extract from a
>>> Drill (1.12 on EMR) cluster memory appears not to adhere to the limits set
>>> by planner.memory.max_query_memor y_per_node.
>>>
>>> The extract query consumes all memory and then crashes drill.
>>>
>>> Running the same query as a create table memory behaves as expected.
>>>
>>> The query complexity is trivial:
>>> select * from view only a single parquet with no calculated fields.
>>>
>>> Has anyone else observed this behavior?
>>>
>>>
>>>
>>>
>>
>
