Re: "org.apache.thrift.transport.TTransportException: Invalid status -128" errors when SASL is enabled

2024-01-11 Thread Austin Hackett 
For the benefit of anyone who comes across this error in future, it was solved 
by adding hive.metastore.sasl.enabled and hive.metastore.kerberos.principal to 
hive-site.xml on the client side, e.g. $SPARK_HOME/conf


> On 8 Jan 2024, at 16:18, Austin Hackett  wrote:
> 
> Hi List
>  
> I'm having an issue where Hive Metastore operations (e.g. show databases) are 
> failing with "org.apache.thrift.transport.TTransportException: Invalid status 
> -128" errors when I enable SASL.
>  
> I am a bit stuck on how to go about troubleshooting this further, and any 
> pointers would be greatly apprecicated...
>  
> Full details as follows:
>  
> - Ubuntu 22.04 & OpenJDK 8u342
> - Unpacked Hive 3.1.3 binary release 
> (https://dlcdn.apache.org/hive/hive-3.1.3/apache-hive-3.1.3-bin.tar.gz) to 
> /opt/hive
> - Unpacked Hadoop 3.1.0 binary release 
> (https://archive.apache.org/dist/hadoop/common/hadoop-3.1.0/hadoop-3.1.0.tar.gz)
>  to /opt/hadoop
> - Created /opt/hive/conf/metastore-site.xml (see below for contents) and 
> copied hdfs-site.xml and core-site.xml from the target HDFS cluster to 
> /opt/hive/conf
> - export HADOOP_HOME=/opt/hadoop
> - export HIVE_HOME=/opt/hive
> - Successfully started the metastore, i.e. hive --service metastore
> - Use a Hive Metastore client to "show databases" and get an error (see below 
> for the associated errors in the HMS log). I get the same error with 
> spark-shell running in local mode and the Python hive-metastore-client 
> (https://pypi.org/project/hive-metastore-client/)
>  
>  
> metastore-site.xml
> ==
> 
>   
> metastore.warehouse.dir
> /user/hive/warehouse
>   
>   
> javax.jdo.option.ConnectionDriverName
> org.postgresql.Driver
>   
>   
> javax.jdo.option.ConnectionURL
> jdbc:postgresql://postgres.example.net:5432/metastore_db 
> 
>   
>   
> javax.jdo.option.ConnectionUserName
> hive
>   
>   
> javax.jdo.option.ConnectionPassword
> password
>   
>   
> metastore.kerberos.principal
> hive/_h...@example.net >
>   
>   
> metastore.kerberos.keytab.file
> /etc/security/keytabs/hive.keytab
>   
>   
> hive.metastore.sasl.enabled
> true
>   
> 
> ==
>  
> HMS log shows that it is able to authenticate using the specified keytab and 
> principle (and I have also checked this manually via kinit command):
>  
> 
> 2024-01-08T13:12:33,463  WARN [main] security.HadoopThriftAuthBridge: 
> Client-facing principal not set. Using server-side setting: 
> hive/_h...@example.net 
> 2024-01-08T13:12:33,464  INFO [main] security.HadoopThriftAuthBridge: Logging 
> in via CLIENT based principal
> 2024-01-08T13:12:33,471 DEBUG [main] security.UserGroupInformation: Hadoop 
> login
> 2024-01-08T13:12:33,472 DEBUG [main] security.UserGroupInformation: hadoop 
> login commit
> 2024-01-08T13:12:33,472 DEBUG [main] security.UserGroupInformation: Using 
> kerberos user: hive/metstore.example@example.net 
> 
> 2024-01-08T13:12:33,472 DEBUG [main] security.UserGroupInformation: Using 
> user: "hive/metstore.example@example.net 
> " with name: 
> hive/metstore.example@example.net 
> 
> 2024-01-08T13:12:33,472 DEBUG [main] security.UserGroupInformation: User 
> entry: "hive/metstore.example@example.net 
> "
> 2024-01-08T13:12:33,472  INFO [main] security.UserGroupInformation: Login 
> successful for user hive/metstore.example@example.net 
>  using keytab file hive.keytab. 
> Keytab auto renewal enabled : false
> 2024-01-08T13:12:33,472  INFO [main] security.HadoopThriftAuthBridge: Logging 
> in via SERVER based principal
> 2024-01-08T13:12:33,480 DEBUG [main] security.UserGroupInformation: Hadoop 
> login
> 2024-01-08T13:12:33,480 DEBUG [main] security.UserGroupInformation: hadoop 
> login commit
> 2024-01-08T13:12:33,480 DEBUG [main] security.UserGroupInformation: Using 
> kerberos user: hive/metstore.example@example.net 
> 
> 2024-01-08T13:12:33,480 DEBUG [main] security.UserGroupInformation: Using 
> user: "hive/metstore.example@example.net 
> " with name: 
> hive/metstore.example@example.net 
> 
> 2024-01-08T13:12:33,480 DEBUG [main] security.UserGroupInformation: User 
> entry: "hive/metstore.example@example.net 
> "
> 2024-01-08T13:12:33,480  INFO [main] security.UserGroupInformation: Login 
> successful for user hive/metstore.example@example.net 
>  using keytab file hive.keytab. 
> Keytab auto renewal enabled : false
> 
>  
>

"org.apache.thrift.transport.TTransportException: Invalid status -128" errors when SASL is enabled

2024-01-08 Thread Austin Hackett 
Hi List
 
I'm having an issue where Hive Metastore operations (e.g. show databases) are 
failing with "org.apache.thrift.transport.TTransportException: Invalid status 
-128" errors when I enable SASL.
 
I am a bit stuck on how to go about troubleshooting this further, and any 
pointers would be greatly apprecicated...
 
Full details as follows:
 
- Ubuntu 22.04 & OpenJDK 8u342
- Unpacked Hive 3.1.3 binary release 
(https://dlcdn.apache.org/hive/hive-3.1.3/apache-hive-3.1.3-bin.tar.gz) to 
/opt/hive
- Unpacked Hadoop 3.1.0 binary release 
(https://archive.apache.org/dist/hadoop/common/hadoop-3.1.0/hadoop-3.1.0.tar.gz)
 to /opt/hadoop
- Created /opt/hive/conf/metastore-site.xml (see below for contents) and copied 
hdfs-site.xml and core-site.xml from the target HDFS cluster to /opt/hive/conf
- export HADOOP_HOME=/opt/hadoop
- export HIVE_HOME=/opt/hive
- Successfully started the metastore, i.e. hive --service metastore
- Use a Hive Metastore client to "show databases" and get an error (see below 
for the associated errors in the HMS log). I get the same error with 
spark-shell running in local mode and the Python hive-metastore-client 
(https://pypi.org/project/hive-metastore-client/)
 
 
metastore-site.xml
==

  
metastore.warehouse.dir
/user/hive/warehouse
  
  
javax.jdo.option.ConnectionDriverName
org.postgresql.Driver
  
  
javax.jdo.option.ConnectionURL
jdbc:postgresql://postgres.example.net:5432/metastore_db 

  
  
javax.jdo.option.ConnectionUserName
hive
  
  
javax.jdo.option.ConnectionPassword
password
  
  
metastore.kerberos.principal
hive/_h...@example.netmailto:hive/_h...@example.net%3c/value>>
  
  
metastore.kerberos.keytab.file
/etc/security/keytabs/hive.keytab
  
  
hive.metastore.sasl.enabled
true
  

==
 
HMS log shows that it is able to authenticate using the specified keytab and 
principle (and I have also checked this manually via kinit command):
 

2024-01-08T13:12:33,463  WARN [main] security.HadoopThriftAuthBridge: 
Client-facing principal not set. Using server-side setting: 
hive/_h...@example.net 
2024-01-08T13:12:33,464  INFO [main] security.HadoopThriftAuthBridge: Logging 
in via CLIENT based principal
2024-01-08T13:12:33,471 DEBUG [main] security.UserGroupInformation: Hadoop login
2024-01-08T13:12:33,472 DEBUG [main] security.UserGroupInformation: hadoop 
login commit
2024-01-08T13:12:33,472 DEBUG [main] security.UserGroupInformation: Using 
kerberos user: hive/metstore.example@example.net 

2024-01-08T13:12:33,472 DEBUG [main] security.UserGroupInformation: Using user: 
"hive/metstore.example@example.net 
" with name: 
hive/metstore.example@example.net 

2024-01-08T13:12:33,472 DEBUG [main] security.UserGroupInformation: User entry: 
"hive/metstore.example@example.net 
"
2024-01-08T13:12:33,472  INFO [main] security.UserGroupInformation: Login 
successful for user hive/metstore.example@example.net 
 using keytab file hive.keytab. 
Keytab auto renewal enabled : false
2024-01-08T13:12:33,472  INFO [main] security.HadoopThriftAuthBridge: Logging 
in via SERVER based principal
2024-01-08T13:12:33,480 DEBUG [main] security.UserGroupInformation: Hadoop login
2024-01-08T13:12:33,480 DEBUG [main] security.UserGroupInformation: hadoop 
login commit
2024-01-08T13:12:33,480 DEBUG [main] security.UserGroupInformation: Using 
kerberos user: hive/metstore.example@example.net 

2024-01-08T13:12:33,480 DEBUG [main] security.UserGroupInformation: Using user: 
"hive/metstore.example@example.net 
" with name: 
hive/metstore.example@example.net 

2024-01-08T13:12:33,480 DEBUG [main] security.UserGroupInformation: User entry: 
"hive/metstore.example@example.net 
"
2024-01-08T13:12:33,480  INFO [main] security.UserGroupInformation: Login 
successful for user hive/metstore.example@example.net 
 using keytab file hive.keytab. 
Keytab auto renewal enabled : false

 
However, when i attempt to "show databases":
 

2024-01-08T13:59:08,068 DEBUG [pool-6-thread-1] security.UserGroupInformation: 
PrivilegedAction [as: hive/metstore.example@example.net 
 
(auth:KERBEROS)][action:org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1@1e655c9
 
]
java.lang.Exception: