Re: Spark Thrift Server java vm problem need help

Mon, 23 Mar 2020 07:41:10 -0700

If -Xmx is bigger then 32g, vm will not to use  UseCompressedOops as default, 
We can see a case, 
If we set spark.driver.memory is 64g, set -XX:+UseCompressedOops in spark.executor.extralJavaOptions, and set SPARK_DAEMON_MEMORY = 6g, 
Use current code , vm will got command like with  -Xmx6g and -XX:+UseCompressedOops , then vm will be -XX:+UseCompressedOops  and use Oops compressed

But since we set spark.driver.memory=64g, our jvm’s max heap size will be 64g,  but we will use compressed Oops ,  Wouldn't that be a problem？

On 03/23/2020 22:32，Sean Owen<sro...@gmail.com> wrote：
I'm still not sure if you are trying to enable it or disable it, and what the issue is?
There is no logic in Spark that sets or disables this flag that I can see.

On Mon, Mar 23, 2020 at 9:27 AM angers.zhu <angers....@gmail.com> wrote:
Hi Sean,

Yea,  I set  -XX:+UseCompressedOops in driver(you can see in command line) and these days, we have more user and I set
spark.driver.memory to 64g, in Non-default VM flags it should be +XX:-UseCompressdOops , but it’s still 
+XX:-UseCompressdOops.

I have find the reason , in SparkSubmitCommandBuilder.buildSparkSubmitCommand, have logic like below

if (isClientMode) {
  // Figuring out where the memory value come from is a little tricky due to precedence.
  // Precedence is observed in the following order:
  // - explicit configuration (setConf()), which also covers --driver-memory cli argument.
  // - properties file.
  // - SPARK_DRIVER_MEMORY env variable
  // - SPARK_MEM env variable
  // - default value (1g)
  // Take Thrift Server as daemon
  String tsMemory =
    isThriftServer(mainClass) ? System.getenv("SPARK_DAEMON_MEMORY") : null;
  String memory = firstNonEmpty(tsMemory, config.get(SparkLauncher.DRIVER_MEMORY),
    System.getenv("SPARK_DRIVER_MEMORY"), System.getenv("SPARK_MEM"), DEFAULT_MEM);
  cmd.add("-Xmx" + memory);
  addOptionString(cmd, driverDefaultJavaOptions);
  addOptionString(cmd, driverExtraJavaOptions);
  mergeEnvPathList(env, getLibPathEnvName(),
    config.get(SparkLauncher.DRIVER_EXTRA_LIBRARY_PATH));
}

For Spark Thrift Server, use SPARK_DAEMON_MEMORY First, it’s really reasonable, I am confused, if spark.driver.memory is bigger then 32g
And SPARK_DAEMON_MEMORY is less then 32g, UseCompressedOops will also be open, it’s right?

If we need to modify this logic for case >32g.


By the way, I meet problem like https://issues.apache.org/jira/browse/SPARK-27097, caused by these strange case.

Thanks

On 03/23/2020 21:43，Sean Owen<sro...@gmail.com> wrote：
I don't think Spark sets UseCompressedOops in any defaults; are you setting it?
It can't be used with heaps >= 32GB. It doesn't seem to cause an error if you set it with large heaps, just a warning.
What's the problem?

On Mon, Mar 23, 2020 at 6:21 AM angers.zhu <angers....@gmail.com> wrote:

    






    

    

        Hi developers,

 These day I meet a strange problem and I can’t find why

When I start a spark thrift server with  spark.driver.memory 64g, then use jdk8/bin/jinfo pid to see vm flags got below information,
In 64g vm, UseCompressedOops should be closed by default, why spark thrift server is -XX: +UseCompressedOops

Non-default VM flags: -XX:CICompilerCount=15 -XX:-CMSClassUnloadingEnabled -XX:CMSFullGCsBeforeCompaction=0 -XX:CMSInitiatingOccupancyFraction=75 -XX:+CMSParallelRemarkEnabled -XX:-ClassUnloading -XX:+DisableExplicitGC -XX:ErrorFile=null -XX:-ExplicitGCInvokesConcurrentAndUnloadsClasses -XX:InitialHeapSize=2116026368 -XX:+ManagementServer -XX:MaxDirectMemorySize=8589934592 -XX:MaxHeapSize=6442450944 -XX:MaxNewSize=2147483648 -XX:MaxTenuringThreshold=6 -XX:MinHeapDeltaBytes=196608 -XX:NewSize=705298432 -XX:OldPLABSize=16 -XX:OldSize=1410727936 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:-TraceClassUnloading -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseFastUnorderedTimeStamps -XX:+UseParNewGC
Command line:  -Xmx6g -Djava.library.path=/home/hadoop/hadoop/lib/native -Djavax.security.auth.useSubjectCredsOnly=false -Dcom.sun.management.jmxremote.port=9021 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -XX:MaxPermSize=1024m -XX:PermSize=256m -XX:MaxDirectMemorySize=8192m -XX:-TraceClassUnloading -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+CMSParallelRemarkEnabled -XX:+DisableExplicitGC -XX:+PrintTenuringDistribution -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=75 -Xnoclassgc -XX:+PrintGCDetails -XX:+PrintGCDateStamps 

Since I am not a professor in VM, hope for some help


    

        
            

        
    

    

      
    
    

        签名由
        网易邮箱大师
        定制
--------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Reply via email to