muhong created SPARK-37821:
------------------------------

             Summary: spark thrift server RDD ID overflow lead sql execute 
failed
                 Key: SPARK-37821
                 URL: https://issues.apache.org/jira/browse/SPARK-37821
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 3.2.0
            Reporter: muhong


this problem will happen in long run spark application,such as thrift server;

as only one SparkContext instance in thrift server driver size,so if the 
concurrency of sql request is large or the sql is too complicate(this will 
create a lot of rdd), the rdd will be generate too fast , the rdd id 
(SparkContext.scala#nextRddId:[https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala]
 )will be consume fast, after a few months the nextRddId will overflow。the 
newRddId will be negative number,but the rdd's block id need to be positive, so 
this will lead a exception"Failed to parse rdd_-2123452330_2 into block ID",so 
can not exchange data during sql execution, and lead sql execute failed



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to