Re: pyspark connect to spark thrift server port

[second_co...@yahoo.com.INVALID]

 
Hello Artemis,   Understand, if i gave hive metastore uri to anyone to connect 
using pyspark. the port 9083 is open for anyone without authentication feature. 
The only way pyspark able to connect to hive is through 9083 and not through 
port 10000. 
    On Friday, October 21, 2022 at 04:06:38 AM GMT+8, Artemis User 
<arte...@dtechspace.com> wrote:  
 
  By default, Spark uses Apache Derby (running in embedded mode with store 
content defined in local files) for hosting the Hive metastore.  You can 
externalize the metastore on a JDBC-compliant database (e.g., PostgreSQL) and 
use the database authentication provided by the database.  The JDBC 
configuration shall be defined in a hive-site.xml file in the Spark conf 
directory.  Please see the metastore admin guide for more details, including an 
init script for setting up your 
metastore(https://cwiki.apache.org/confluence/display/Hive/AdminManual+Metastore+3.0+Administration).
 
 
 On 10/20/22 4:31 AM, second_co...@yahoo.com.INVALID wrote:
  
 
 Currently my pyspark code able to connect to hive metastore at port 9083. 
However using this approach i can't put in-place any security mechanism like 
LDAP and sql authentication control. Is there anyway to connect from pyspark to 
spark thrift server on port 10000 without exposing hive metastore url to the 
pyspark ? I would like to authenticate the user before allow to execute spark 
sql, and user should only allow to query from databases,tables that they have 
the access.
  
  
  
  Thank you, comet