[
https://issues.apache.org/jira/browse/HIVE-18214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16312261#comment-16312261
]
Aihua Xu commented on HIVE-18214:
---------------------------------
[~stakiar] Try to understand the issue. So when the one test finishes, rpc is
closing and it will close RemoteDriver to stop the SparkContext, but since it
happens asynchronously, we don't know when it really shutdowns?
One thought: maybe we should add the logic to always make sure there is only
JavaSparkContext instance created in one JVM. If there is one existing and we
try to create a new one, we can shutdown the existing one and create a new one.
> Flaky test: TestSparkClient
> ---------------------------
>
> Key: HIVE-18214
> URL: https://issues.apache.org/jira/browse/HIVE-18214
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Attachments: HIVE-18214.1.patch
>
>
> Looks like there is a race condition in {{TestSparkClient#runTest}}. The test
> creates a {{RemoteDriver}} in memory, which creates a {{JavaSparkContext}}. A
> new {{JavaSparkContext}} is created for each test that is run. There is a
> race condition where the {{RemoteDriver}} isn't given enough time to
> shutdown, so when the next test starts running it creates another
> {{JavaSparkContext}} which causes an exception like
> {{org.apache.spark.SparkException: Only one SparkContext may be running in
> this JVM (see SPARK-2243)}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
