[ https://issues.apache.org/jira/browse/SPARK-23872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16786454#comment-16786454 ]
Udbhav Agrawal commented on SPARK-23872: ---------------------------------------- Hi [~chelsa] , you have to close the spark_1 session to clear the active sparkcontext, otherwise new configuration won't be loaded. So call spark_1.stop() > Can not connect to another metastore uri using two Spark sessions > ----------------------------------------------------------------- > > Key: SPARK-23872 > URL: https://issues.apache.org/jira/browse/SPARK-23872 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.3.0 > Environment: OS :CentOS release 6.8 (Final) > JAVA : build 1.8.0_101-b13 > SPARK : 2.3.0 > > Reporter: Chan Min Park > Priority: Major > > In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session > metasore information is used when the second session is run > $################## Run Source Code $################## > val spark_1 = SparkSession.builder() > .enableHiveSupport() > .config("hive.metastore.uris", "thrift://HOST_A:9083") > .getOrCreate() > spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() > SparkSession.clearActiveSession() > SparkSession.clearDefaultSession() > val spark_2 = SparkSession.builder() > .enableHiveSupport() > .config("hive.metastore.uris", "thrift://HOST_B:9083") > .getOrCreate() > spark_2.sql("SELECT B_FIELD FROM TABLE_B").show() > $################## Run info result in spark 2.1.0 $################## > ...... > INFO metastore: Trying to connect to metastore with URI > thrift://*{color:#d04437}HOST_A{color}*:9083 > ...... > INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took > 0.807905 s > +------------ + > / A_FIELD / > --------------- > / A / > ---------------- > ...... > INFO metastore: Trying to connect to metastore with URI > thrift://*{color:#d04437}HOST_B{color}*:9083 > INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took > 0.807905 s > +------------ + > / B_FIELD / > --------------- > / B / > ---------------- > ...... > $################## Run info result in spark 2.3.0 $################## > ...... > INFO metastore: Trying to connect to metastore with URI > thrift://*{color:#d04437}HOST_A{color}*:9083 > ...... > INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took > 0.807905 s > +------------ + > / A_FIELD / > --------------- > / A / > ---------------- > ...... > INFO metastore: Trying to connect to metastore with URI > thrift://*{color:#d04437}HOST_A{color}*:9083 > ...... > Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or > view not found: `default`.`TABLE_B`; line 1 pos 19; > ************************************************************************************** -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org