Re: High CPU usage on Hbase region Server with GlobalMemoryManager warnings

[Pedro Boado]

Maybe the warnings are not the cause but a consequence (gc calls finalize()
and not the other way around)

Any details on memory usage? G1? Non full Vs full gc ratio, average freed
memory... Did any gc run in the 3rd RS?

Memory percentage assigned to memstores? You have an average memory
assigned of 120MB per region assuming 1 column family per table, and HBase
default memstore size  is already 256mb. You might just being running oom.

Read/write pattern? Is there an intensive write?

In my experience these kind of issues are caused by a bad HBase memory
tuning.






On 1 Feb 2018 02:41, "Jins George"  wrote:

Hi,

On analyzing a prod issue of High CPU usage on Hbase Region server, I came
across warning messages from region server logs complaining about Orphaned
chunk of memory.

2018-01-30 19:16:31,565 WARN
org.apache.phoenix.memory.GlobalMemoryManager: Orphaned chunk of
104000 bytes found during finalize
2018-01-30 19:16:31,565 WARN
org.apache.phoenix.memory.GlobalMemoryManager: Orphaned chunk of
104000 bytes found during finalize


The high CPU usage looks like due to garbage collection and it lasted for
almost 6 hours.  And throughout 6 hours, region server logs had these
warning messages logged.

Cluster Details:
4 node( 1 master + 3 slaves)  cdh cluster
Hbase version 1.2
Phoenix version 4.7
Region Server Heap : 4G
Total Regions: ~135
Total tables : ~35

Out of 3 region servers, 2 of them had the warning logs and both suffered
high CPU. Third region server nither had High CPU nor the warning logs. Any
idea why these messages are logged and can that trigger continuous GC ?

Before this issue started( or around the same time) huge application log
files were copied to HDFS by Yarn.. But can't think of that causing issue
on Hbase Region  server.

Any help is appreciated.

Thanks,
Jins George

Re: [External] High CPU usage on Hbase region Server with GlobalMemoryManager warnings

[Durmus, Yunus]

Hi,

Thursday I am in parental leave, I will respond you on Friday.
For urgent matters, feel free to call me directly.

Best regards
--yunus



This message is for the designated recipient only and may contain privileged, 
proprietary, or otherwise confidential information. If you have received it in 
error, please notify the sender immediately and delete the original. Any other 
use of the e-mail by you is prohibited. Where allowed by local law, electronic 
communications with Accenture and its affiliates, including e-mail and instant 
messaging (including content), may be scanned by our systems for the purposes 
of information security and assessment of internal compliance with Accenture 
policy.
__

www.accenture.com

High CPU usage on Hbase region Server with GlobalMemoryManager warnings

[Jins George]

Hi,

On analyzing a prod issue of High CPU usage on Hbase Region server, I 
came across warning messages from region server logs complaining about 
Orphaned chunk of memory.


2018-01-30 19:16:31,565 WARN org.apache.phoenix.memory.GlobalMemoryManager: 
Orphaned chunk of 104000 bytes found during finalize
2018-01-30 19:16:31,565 WARN org.apache.phoenix.memory.GlobalMemoryManager: 
Orphaned chunk of 104000 bytes found during finalize


The high CPU usage looks like due to garbage collection and it lasted 
for almost 6 hours.  And throughout 6 hours, region server logs had 
these warning messages logged.


Cluster Details:
4 node( 1 master + 3 slaves)  cdh cluster
Hbase version 1.2
Phoenix version 4.7
Region Server Heap : 4G
Total Regions: ~135
Total tables : ~35

Out of 3 region servers, 2 of them had the warning logs and both 
suffered high CPU. Third region server nither had High CPU nor the 
warning logs. Any idea why these messages are logged and can that 
trigger continuous GC ?


Before this issue started( or around the same time) huge application log 
files were copied to HDFS by Yarn.. But can't think of that causing 
issue on Hbase Region  server.


Any help is appreciated.

Thanks,
Jins George

Re: Spark 2.1.1 (Scala 2.11.8) write to Phoenix 4.7 (HBase 1.1.2)

[Josh Mahonin]

Hi,

As per https://phoenix.apache.org/phoenix_spark.html, Apache Phoenix is
compiled against Spark2 only in versions 4.10 and above. If you must use
Phoenix 4.7 against Spark 2.x, you may need to apply PHOENIX- yourself:

https://github.com/apache/phoenix/commit/a0e5efcec5a1a732b2dce9794251242c3d66eea6#diff-600376dffeb79835ede4a0b285078036

Note that if you're using a vendor-provided Phoenix distribution, there may
already be support for the Spark version you're using. Please follow-up
with them if that's the case.

Best,

Josh




On Tue, Jan 30, 2018 at 10:06 AM, Margusja  wrote:

> Also I see that Logging is moved to internal/Logging. But is there package
> for my environment I can use?
>
> Margus
>
>
> On 30 Jan 2018, at 17:00, Margusja  wrote:
>
> Hi
>
> Followed page (https://phoenix.apache.org/phoenix_spark.html  phoenix.apache.org/phoenix_spark.html>) and trying to save to phoenix.
>
> Using spark-1.6.3 it is successful but using spark-2.1.1 it is not.
> First error I am getting using spark-2.1.1 is that:
>
> Error:scalac: missing or invalid dependency detected while loading class
> file 'ProductRDDFunctions.class'.
> Could not access type Logging in package org.apache.spark,
> because it (or its dependencies) are missing. Check your build definition
> for
> missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see
> the problematic classpath.)
> A full rebuild may help if 'ProductRDDFunctions.class' was compiled
> against an incompatible version of org.apache.spark.
>
> I can see that Logging is removed after 1.6.3 and does not exist in 2.1.1.
>
> What are my options?
>
> Br
> Margus
>
>
>