Something like this in Python
from pyspark.sql import SparkSession
# Configure Spark Session with JDBC URLs
spark_conf = SparkConf() \
.setAppName("SparkCatalogMultipleSources") \
.set("hive.metastore.uris",
"thrift://hive1-metastore:9080,thrift://hive2-metastore:9080")
jdbc_urls = ["jdbc:hive2://hive1-jdbc:10000",
"jdbc:hive2://hive2-jdbc:10000"]
mysql_jdbc_url = "jdbc:mysql://mysql-host:3306/mysql_database"
spark = SparkSession.builder \
.config(spark_conf) \
.enableHiveSupport() \
.getOrCreate()
# Accessing tables from Hive1, Hive2, and MySQL
spark.sql("SELECT * FROM hive1.table1").show()
spark.sql("SELECT * FROM hive2.table2").show()
spark.sql("SELECT * FROM mysql.table1").show()
# Optional: Create temporary views for easier joining (if needed)
spark.sql("CREATE TEMPORARY VIEW hive1_table1 AS SELECT * FROM
hive1.table1")
spark.sql("CREATE TEMPORARY VIEW hive2_table2 AS SELECT * FROM
hive2.table2")
spark.sql("CREATE TEMPORARY VIEW mysql_table1 AS SELECT * FROM
mysql.table1")
HTH
Mich Talebzadeh,
Technologist | Architect | Data Engineer | Generative AI | FinCrime
London
United Kingdom
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* The information provided is correct to the best of my
knowledge but of course cannot be guaranteed . It is essential to note
that, as with any advice, quote "one test result is worth one-thousand
expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von
Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)".
On Fri, 24 May 2024 at 09:41, 志阳 <[email protected]> wrote:
> I have two clusters hive1 and hive2, as well as a MySQL database. Can I
> use Spark Catalog for registration, but can I only use one catalog at a
> time? Can multiple catalogs be joined across databases.
> select * from
> hive1.table1 join hive2.table2 join mysql.table1
> where ....
>
> ------------------------------
> 志阳
> [email protected]
>
> <https://wx.mail.qq.com/home/index?t=readmail_businesscard_midpage&nocheck=true&name=%E5%BF%97%E9%98%B3&icon=https%3A%2F%2Fthirdqq.qlogo.cn%2Fg%3Fb%3Doidb%26k%3DHWGP96VD8rBRZI1WK8ibztQ%26s%3D0&mail=308027245%40qq.com&code=pIwrzsvlK4y3scaRuDSCa1GM8dkNk7pp9uoMCQhamZ_pD_ZyxuILhRspClLePCouK3NCv_bjUAZeqemuvmns0w>
>
>
