Re: Question about memory chunk and waiting for memory to become free

[lars hofhansl]

Thanks James.
I'll file a jira.

  From: James Taylor 
 To: dev@phoenix.apache.org; lars hofhansl  
 Sent: Monday, September 4, 2017 9:11 AM
 Subject: Re: Question about memory chunk and waiting for memory to become free
   
Sure, Lars. As long as we throw an exception that'll cause a retry, it
seems fine to just fail it. We'll need some kind of synchronization around
the read/write of available memory member variable (or we can just use an
atomic long). Please update the docs here and indicate that
phoenix.query.maxGlobalMemoryWaitMs is deprecated as of 4.12:
http://phoenix.apache.org/tuning.html

On Sun, Sep 3, 2017 at 11:28 AM lars hofhansl  wrote:

> Hi,
> just looking at MemoryChunk for various reasons.
> What I did not realize is that when no memory is currently available the
> implementation waits for memory to become available.The default wait time
> is 10s.
>
> This seems very dangerous to me. This can hang HBase handler threads at
> various points, especially when Phoenix is "busy" (I realize we're not
> talking about real memory here, but the effect is still that the HBase
> server threads would stall).
> HBase was designed with the assumption that individual RPCs do not tie up
> a handler thread for long. (The worst offender is a scan RPC that could
> scan up to a region's worth if a highly selective filter was passed). It's
> also possible that this is mostly used in asynchronous processing, although
> a quick scan of the code suggests that's not the case.
>
> It's possible that I'm missing something (in which case someone enlighten
> me please). I feel with wait time should be set to 0. I.e. we check whether
> there's memory available right now, if not we just fail the request and
> free the handler thread. I'd even argue we can remove the
> wait/synchronization code completely from MemoryChunk.
>
> Comments?
> -- Lars
>


   

Re: Question about memory chunk and waiting for memory to become free

[James Taylor]

Sure, Lars. As long as we throw an exception that'll cause a retry, it
seems fine to just fail it. We'll need some kind of synchronization around
the read/write of available memory member variable (or we can just use an
atomic long). Please update the docs here and indicate that
phoenix.query.maxGlobalMemoryWaitMs is deprecated as of 4.12:
http://phoenix.apache.org/tuning.html

On Sun, Sep 3, 2017 at 11:28 AM lars hofhansl  wrote:

> Hi,
> just looking at MemoryChunk for various reasons.
> What I did not realize is that when no memory is currently available the
> implementation waits for memory to become available.The default wait time
> is 10s.
>
> This seems very dangerous to me. This can hang HBase handler threads at
> various points, especially when Phoenix is "busy" (I realize we're not
> talking about real memory here, but the effect is still that the HBase
> server threads would stall).
> HBase was designed with the assumption that individual RPCs do not tie up
> a handler thread for long. (The worst offender is a scan RPC that could
> scan up to a region's worth if a highly selective filter was passed). It's
> also possible that this is mostly used in asynchronous processing, although
> a quick scan of the code suggests that's not the case.
>
> It's possible that I'm missing something (in which case someone enlighten
> me please). I feel with wait time should be set to 0. I.e. we check whether
> there's memory available right now, if not we just fail the request and
> free the handler thread. I'd even argue we can remove the
> wait/synchronization code completely from MemoryChunk.
>
> Comments?
> -- Lars
>

Question about memory chunk and waiting for memory to become free

[lars hofhansl]

Hi,
just looking at MemoryChunk for various reasons.
What I did not realize is that when no memory is currently available the 
implementation waits for memory to become available.The default wait time is 
10s.

This seems very dangerous to me. This can hang HBase handler threads at various 
points, especially when Phoenix is "busy" (I realize we're not talking about 
real memory here, but the effect is still that the HBase server threads would 
stall).
HBase was designed with the assumption that individual RPCs do not tie up a 
handler thread for long. (The worst offender is a scan RPC that could scan up 
to a region's worth if a highly selective filter was passed). It's also 
possible that this is mostly used in asynchronous processing, although a quick 
scan of the code suggests that's not the case.

It's possible that I'm missing something (in which case someone enlighten me 
please). I feel with wait time should be set to 0. I.e. we check whether 
there's memory available right now, if not we just fail the request and free 
the handler thread. I'd even argue we can remove the wait/synchronization code 
completely from MemoryChunk.

Comments?
-- Lars