I guess there are some confusions here between the metastore and the
actual Hive database. Spark (as well as Apache Hive) requires two
databases for Hive DB operations. Metastore is used for storing
metadata only (e.g., schema info), whereas the actual Hive database,
accessible through Thrift server, is used for applications. The reason
why Hive needs its metadata stored separately as a server is because for
distributed database operations.
My previous message referred to how to secure the metastore database,
not the actual Hive tables. Looks like you are looking for how to
secure access to Hive not metastore (metastore isn't used by general
users), and your current configuration wasn't set up with the right user
access control. Hive actually supports role-based access model just
like other RDBMS. You may refer to the Hive admin guide for more
details
(https://cwiki.apache.org/confluence/display/Hive/SQL+Standard+Based+Hive+Authorization).
You can use beeline or SQL scripts via beeline to set user privileges
and roles.
On 10/21/22 1:27 AM, second_co...@yahoo.com.INVALID wrote:
Hello Artemis,
Understand, if i gave hive metastore uri to anyone to connect using
pyspark. the port 9083 is open for anyone without authentication
feature. The only way pyspark able to connect to hive is through 9083
and not through port 10000.
On Friday, October 21, 2022 at 04:06:38 AM GMT+8, Artemis User
<arte...@dtechspace.com> wrote:
By default, Spark uses Apache Derby (running in embedded mode with
store content defined in local files) for hosting the Hive metastore.
You can externalize the metastore on a JDBC-compliant database (e.g.,
PostgreSQL) and use the database authentication provided by the
database. The JDBC configuration shall be defined in a hive-site.xml
file in the Spark conf directory. Please see the metastore admin
guide for more details, including an init script for setting up your
metastore
(https://cwiki.apache.org/confluence/display/Hive/AdminManual+Metastore+3.0+Administration
<https://cwiki.apache.org/confluence/display/Hive/AdminManual+Metastore+3.0+Administration>).
On 10/20/22 4:31 AM, second_co...@yahoo.com.INVALID
<mailto:second_co...@yahoo.com.INVALID> wrote:
Currently my pyspark code able to connect to hive metastore at port
9083. However using this approach i can't put in-place any security
mechanism like LDAP and sql authentication control. Is there anyway to
connect from pyspark to spark thrift server on port 10000 without
exposing hive metastore url to the pyspark ? I would like to
authenticate the user before allow to execute spark sql, and user
should only allow to query from databases,tables that they have the
access.
Thank you,
comet
