[ https://issues.apache.org/jira/browse/SPARK-23872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Park Chan Min updated SPARK-23872: ---------------------------------- Description: In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session metasore information is used when the second session is run ************************** Run Source Code ************************** val spark_1 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_A:9083") .getOrCreate() spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() SparkSession.clearActiveSession() SparkSession.clearDefaultSession() val spark_2 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_B:9083") .getOrCreate() spark_2.sql("SELECT B_FIELD FROM TABLE_B").show() ################### Run info result in spark 2.1.0 ...... INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 ...... INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +------------ + / A_FIELD / --------------- / A / ---------------- ...... INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_B{color}*:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +------------ + / B_FIELD / --------------- / B / ---------------- ...... ...... INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 ...... INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +------------ + / A_FIELD / --------------- / A / ---------------- ...... INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 ...... Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: `default`.`TABLE_B`; line 1 pos 19; ************************************************************************************** was: In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session metasore information is used when the second session is run ************************** Run Source Code ************************** val spark_1 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_A:9083") .getOrCreate() spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() SparkSession.clearActiveSession() SparkSession.clearDefaultSession() val spark_2 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_B:9083") .getOrCreate() spark_2.sql("SELECT B_FIELD FROM TABLE_B").show() Run info result in spark 2.1.0 ...... INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 ...... INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +------------ + / A_FIELD / --------------- / A / ---------------- ...... INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_B{color}*:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +------------ + / B_FIELD / --------------- / B / ---------------- ...... ...... INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 ...... INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +------------ + / A_FIELD / --------------- / A / ---------------- ...... INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 ...... Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: `default`.`TABLE_B`; line 1 pos 19; ************************************************************************************** > Can not connect to another metastore uri using two Spark sessions > ----------------------------------------------------------------- > > Key: SPARK-23872 > URL: https://issues.apache.org/jira/browse/SPARK-23872 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.3.0 > Environment: OS :CentOS release 6.8 (Final) > JAVA : build 1.8.0_101-b13 > SPARK : 2.3.0 > > Reporter: Park Chan Min > Priority: Major > > In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session > metasore information is used when the second session is run > ************************** Run Source Code ************************** > val spark_1 = SparkSession.builder() > .enableHiveSupport() > .config("hive.metastore.uris", "thrift://HOST_A:9083") > .getOrCreate() > spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() > SparkSession.clearActiveSession() > SparkSession.clearDefaultSession() > val spark_2 = SparkSession.builder() > .enableHiveSupport() > .config("hive.metastore.uris", "thrift://HOST_B:9083") > .getOrCreate() > spark_2.sql("SELECT B_FIELD FROM TABLE_B").show() > ################### Run info result in spark 2.1.0 > ...... > INFO metastore: Trying to connect to metastore with URI > thrift://*{color:#d04437}HOST_A{color}*:9083 > ...... > INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took > 0.807905 s > +------------ + > / A_FIELD / > --------------- > / A / > ---------------- > ...... > INFO metastore: Trying to connect to metastore with URI > thrift://*{color:#d04437}HOST_B{color}*:9083 > INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took > 0.807905 s > +------------ + > / B_FIELD / > --------------- > / B / > ---------------- > ...... > ...... > INFO metastore: Trying to connect to metastore with URI > thrift://*{color:#d04437}HOST_A{color}*:9083 > ...... > INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took > 0.807905 s > +------------ + > / A_FIELD / > --------------- > / A / > ---------------- > ...... > INFO metastore: Trying to connect to metastore with URI > thrift://*{color:#d04437}HOST_A{color}*:9083 > ...... > Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or > view not found: `default`.`TABLE_B`; line 1 pos 19; > ************************************************************************************** -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org