回复：【flink sql-client 读写 Kerberos认证的hive】
好的，谢谢！ ---原始邮件--- 发件人: "Rui Li"https://issues.apache.org/jira/browse/FLINK-20913 有关了，这个issue是1.12.2修复的，可以升级一下试试。 On Mon, Mar 8, 2021 at 2:15 PM guoyb <861277...@qq.com> wrote: > 您好！ > hive.metastore.sasl.enabled 是true > > > 启动sql client的时候，可以正常读取到认证信息，并读取metastore的表名。 > > > 读和写，认证就失败了。 > > > > ---原始邮件--- > 发件人: "Rui Li"
Re: 【flink sql-client 读写 Kerberos认证的hive】
那应该就是跟https://issues.apache.org/jira/browse/FLINK-20913 有关了，这个issue是1.12.2修复的，可以升级一下试试。 On Mon, Mar 8, 2021 at 2:15 PM guoyb <861277...@qq.com> wrote: > 您好！ > hive.metastore.sasl.enabled 是true > > > 启动sql client的时候，可以正常读取到认证信息，并读取metastore的表名。 > > > 读和写，认证就失败了。 > > > > ---原始邮件--- > 发件人: "Rui Li" 发送时间: 2021年3月8日(周一) 中午12:12 > 收件人: "user-zh" 主题: Re: 【flink sql-client 读写 Kerberos认证的hive】 > > > Hi， > > > 从你发的stacktrace来看，走到了set_ugi方法说明client认为server没有开启kerberos。确认一下你HiveCatalog这边指定的hive-site.xml是否配置正确呢，像hive.metastore.sasl.enabled是不是设置成true了？ > > On Sun, Mar 7, 2021 at 5:49 PM 861277...@qq.com <861277...@qq.com> > wrote: > > > 环境： > > flink1.12.1 > > hive2.1.0 > > CDH6.2.0 > > > > > > 【问题描述】 > > 在没开启Kerberos认证时，可以正常读写hive表 > > > > 开启Kerberos认证后， > > 启动时可以正常读取到hive metastore的元数据信息，读写不了表。 > > > > > > 【sql-client.sh embedded】 > > Flink SQL> show tables; > > dimension_table > > dimension_table1 > > test > > > > > > Flink SQL> select * from test; > > [ERROR] Could not execute SQL statement. Reason: > > org.apache.flink.connectors.hive.FlinkHiveException: Failed to > collect all > > partitions from hive metaStore > > > > > > 【完整日志 > > > /opt/cloudera/parcels/FLINK-1.12.1-BIN-SCALA_2.11/lib/flink/log/flink-root-sql-client-cdh6.com.log】 > > > > 2021-03-07 10:29:18.776 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > > - Trying to connect to localhost/127.0.0.1:6123 > > 2021-03-07 10:29:18.777 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > > - Failed to connect from address 'cdh6.com/192.168.31.10': Connection > > refused (Connection refused) > > 2021-03-07 10:29:18.778 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > > - Failed to connect from address '/127.0.0.1': Connection refused > > (Connection refused) > > 2021-03-07 10:29:18.778 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > > - Failed to connect from address > '/fe80:0:0:0:20c:29ff:fea1:6d6b%ens33': > > Network is unreachable (connect failed) > > 2021-03-07 10:29:18.778 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > > - Failed to connect from address '/192.168.31.10': Connection refused > > (Connection refused) > > 2021-03-07 10:29:18.779 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > > - Failed to connect from address '/0:0:0:0:0:0:0:1%lo': Network is > > unreachable (connect failed) > > 2021-03-07 10:29:18.779 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > > - Failed to connect from address '/127.0.0.1': Connection refused > > (Connection refused) > > 2021-03-07 10:29:18.779 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > > - Failed to connect from address > '/fe80:0:0:0:20c:29ff:fea1:6d6b%ens33': > > Network is unreachable (connect failed) > > 2021-03-07 10:29:18.779 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > > - Failed to connect from address '/192.168.31.10': Connection refused > > (Connection refused) > > 2021-03-07 10:29:18.780 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > > - Failed to connect from address '/0:0:0:0:0:0:0:1%lo': Network is > > unreachable (connect failed) > > 2021-03-07 10:29:18.780 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > > - Failed to connect from address '/127.0.0.1': Connection refused > > (Connection refused) > > 2021-03-07 10:29:18.780 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > > - Could not connect. Waiting for 1600 msecs before next attempt > > 2021-03-07 10:29:20.381 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > > - Trying to connect to localhost/127.0.0.1:6123 > > 2021-03-07 10:29:20.381 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > > - Failed to connect from address 'cdh6.com/192.168.31.10': Connection > > refused (Connection refused) > > 2021-03-07 10:29:20.382 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > > - Failed to connect from address '/127
回复：【flink sql-client 读写 Kerberos认证的hive】
您好！ hive.metastore.sasl.enabled 是true 启动sql client的时候，可以正常读取到认证信息，并读取metastore的表名。 读和写，认证就失败了。 ---原始邮件--- 发件人: "Rui Li"
Re: 【flink sql-client 读写 Kerberos认证的hive】
也可能是需要这个fix：https://issues.apache.org/jira/browse/FLINK-20913 On Mon, Mar 8, 2021 at 12:11 PM Rui Li wrote: > Hi， > > > 从你发的stacktrace来看，走到了set_ugi方法说明client认为server没有开启kerberos。确认一下你HiveCatalog这边指定的hive-site.xml是否配置正确呢，像hive.metastore.sasl.enabled是不是设置成true了？ > > On Sun, Mar 7, 2021 at 5:49 PM 861277...@qq.com <861277...@qq.com> wrote: > >> 环境： >> flink1.12.1 >> hive2.1.0 >> CDH6.2.0 >> >> >> 【问题描述】 >> 在没开启Kerberos认证时，可以正常读写hive表 >> >> 开启Kerberos认证后， >> 启动时可以正常读取到hive metastore的元数据信息，读写不了表。 >> >> >> 【sql-client.sh embedded】 >> Flink SQL> show tables; >> dimension_table >> dimension_table1 >> test >> >> >> Flink SQL> select * from test; >> [ERROR] Could not execute SQL statement. Reason: >> org.apache.flink.connectors.hive.FlinkHiveException: Failed to collect >> all partitions from hive metaStore >> >> >> 【完整日志 >> /opt/cloudera/parcels/FLINK-1.12.1-BIN-SCALA_2.11/lib/flink/log/flink-root-sql-client-cdh6.com.log】 >> >> 2021-03-07 10:29:18.776 [main] INFO >> org.apache.flink.runtime.net.ConnectionUtils >> - Trying to connect to localhost/127.0.0.1:6123 >> 2021-03-07 10:29:18.777 [main] INFO >> org.apache.flink.runtime.net.ConnectionUtils >> - Failed to connect from address 'cdh6.com/192.168.31.10': Connection >> refused (Connection refused) >> 2021-03-07 10:29:18.778 [main] INFO >> org.apache.flink.runtime.net.ConnectionUtils >> - Failed to connect from address '/127.0.0.1': Connection refused >> (Connection refused) >> 2021-03-07 10:29:18.778 [main] INFO >> org.apache.flink.runtime.net.ConnectionUtils >> - Failed to connect from address '/fe80:0:0:0:20c:29ff:fea1:6d6b%ens33': >> Network is unreachable (connect failed) >> 2021-03-07 10:29:18.778 [main] INFO >> org.apache.flink.runtime.net.ConnectionUtils >> - Failed to connect from address '/192.168.31.10': Connection refused >> (Connection refused) >> 2021-03-07 10:29:18.779 [main] INFO >> org.apache.flink.runtime.net.ConnectionUtils >> - Failed to connect from address '/0:0:0:0:0:0:0:1%lo': Network is >> unreachable (connect failed) >> 2021-03-07 10:29:18.779 [main] INFO >> org.apache.flink.runtime.net.ConnectionUtils >> - Failed to connect from address '/127.0.0.1': Connection refused >> (Connection refused) >> 2021-03-07 10:29:18.779 [main] INFO >> org.apache.flink.runtime.net.ConnectionUtils >> - Failed to connect from address '/fe80:0:0:0:20c:29ff:fea1:6d6b%ens33': >> Network is unreachable (connect failed) >> 2021-03-07 10:29:18.779 [main] INFO >> org.apache.flink.runtime.net.ConnectionUtils >> - Failed to connect from address '/192.168.31.10': Connection refused >> (Connection refused) >> 2021-03-07 10:29:18.780 [main] INFO >> org.apache.flink.runtime.net.ConnectionUtils >> - Failed to connect from address '/0:0:0:0:0:0:0:1%lo': Network is >> unreachable (connect failed) >> 2021-03-07 10:29:18.780 [main] INFO >> org.apache.flink.runtime.net.ConnectionUtils >> - Failed to connect from address '/127.0.0.1': Connection refused >> (Connection refused) >> 2021-03-07 10:29:18.780 [main] INFO >> org.apache.flink.runtime.net.ConnectionUtils >> - Could not connect. Waiting for 1600 msecs before next attempt >> 2021-03-07 10:29:20.381 [main] INFO >> org.apache.flink.runtime.net.ConnectionUtils >> - Trying to connect to localhost/127.0.0.1:6123 >> 2021-03-07 10:29:20.381 [main] INFO >> org.apache.flink.runtime.net.ConnectionUtils >> - Failed to connect from address 'cdh6.com/192.168.31.10': Connection >> refused (Connection refused) >> 2021-03-07 10:29:20.382 [main] INFO >> org.apache.flink.runtime.net.ConnectionUtils >> - Failed to connect from address '/127.0.0.1': Connection refused >> (Connection refused) >> 2021-03-07 10:29:20.383 [main] INFO >> org.apache.flink.runtime.net.ConnectionUtils >> - Failed to connect from address '/fe80:0:0:0:20c:29ff:fea1:6d6b%ens33': >> Network is unreachable (connect failed) >> 2021-03-07 10:29:20.383 [main] INFO >> org.apache.flink.runtime.net.ConnectionUtils >> - Failed to connect from address '/192.168.31.10': Connection refused >> (Connection refused) >> 2021-03-07 10:29:20.383 [main] INFO >> org.apache.flink.runtime.net.ConnectionUtils >> - Failed to connect from address '/0:0:0:0:0:0:0:1%lo': Network is >> unreachable (connect failed) >> 2021-03-07 10:29:20.383 [main] INFO >> org.apache.flink.runtime.net.ConnectionUtils >> - Failed to connect from address '/127.0.0.1': Connection refused >> (Connection refused) >> 2021-03-07 10:29:20.384 [main] INFO >> org.apache.flink.runtime.net.ConnectionUtils >> - Failed to connect from address '/fe80:0:0:0:20c:29ff:fea1:6d6b%ens33': >> Network is unreachable (connect failed) >> 2021-03-07 10:29:20.384 [main] INFO >> org.apache.flink.runtime.net.ConnectionUtils >> - Failed to connect from address '/192.168.31.10': Connection refused >> (Connection refused) >> 2021-03-07 10:29:20.384 [main] INFO >> org.apache.flink.runtime.net.ConnectionUtils >> - Failed to connect from address '/0:0:0:0:0:0:0:1%lo': Network
Re: 【flink sql-client 读写 Kerberos认证的hive】
Hi， 从你发的stacktrace来看，走到了set_ugi方法说明client认为server没有开启kerberos。确认一下你HiveCatalog这边指定的hive-site.xml是否配置正确呢，像hive.metastore.sasl.enabled是不是设置成true了？ On Sun, Mar 7, 2021 at 5:49 PM 861277...@qq.com <861277...@qq.com> wrote: > 环境： > flink1.12.1 > hive2.1.0 > CDH6.2.0 > > > 【问题描述】 > 在没开启Kerberos认证时，可以正常读写hive表 > > 开启Kerberos认证后， > 启动时可以正常读取到hive metastore的元数据信息，读写不了表。 > > > 【sql-client.sh embedded】 > Flink SQL> show tables; > dimension_table > dimension_table1 > test > > > Flink SQL> select * from test; > [ERROR] Could not execute SQL statement. Reason: > org.apache.flink.connectors.hive.FlinkHiveException: Failed to collect all > partitions from hive metaStore > > > 【完整日志 > /opt/cloudera/parcels/FLINK-1.12.1-BIN-SCALA_2.11/lib/flink/log/flink-root-sql-client-cdh6.com.log】 > > 2021-03-07 10:29:18.776 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > - Trying to connect to localhost/127.0.0.1:6123 > 2021-03-07 10:29:18.777 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > - Failed to connect from address 'cdh6.com/192.168.31.10': Connection > refused (Connection refused) > 2021-03-07 10:29:18.778 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > - Failed to connect from address '/127.0.0.1': Connection refused > (Connection refused) > 2021-03-07 10:29:18.778 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > - Failed to connect from address '/fe80:0:0:0:20c:29ff:fea1:6d6b%ens33': > Network is unreachable (connect failed) > 2021-03-07 10:29:18.778 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > - Failed to connect from address '/192.168.31.10': Connection refused > (Connection refused) > 2021-03-07 10:29:18.779 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > - Failed to connect from address '/0:0:0:0:0:0:0:1%lo': Network is > unreachable (connect failed) > 2021-03-07 10:29:18.779 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > - Failed to connect from address '/127.0.0.1': Connection refused > (Connection refused) > 2021-03-07 10:29:18.779 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > - Failed to connect from address '/fe80:0:0:0:20c:29ff:fea1:6d6b%ens33': > Network is unreachable (connect failed) > 2021-03-07 10:29:18.779 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > - Failed to connect from address '/192.168.31.10': Connection refused > (Connection refused) > 2021-03-07 10:29:18.780 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > - Failed to connect from address '/0:0:0:0:0:0:0:1%lo': Network is > unreachable (connect failed) > 2021-03-07 10:29:18.780 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > - Failed to connect from address '/127.0.0.1': Connection refused > (Connection refused) > 2021-03-07 10:29:18.780 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > - Could not connect. Waiting for 1600 msecs before next attempt > 2021-03-07 10:29:20.381 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > - Trying to connect to localhost/127.0.0.1:6123 > 2021-03-07 10:29:20.381 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > - Failed to connect from address 'cdh6.com/192.168.31.10': Connection > refused (Connection refused) > 2021-03-07 10:29:20.382 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > - Failed to connect from address '/127.0.0.1': Connection refused > (Connection refused) > 2021-03-07 10:29:20.383 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > - Failed to connect from address '/fe80:0:0:0:20c:29ff:fea1:6d6b%ens33': > Network is unreachable (connect failed) > 2021-03-07 10:29:20.383 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > - Failed to connect from address '/192.168.31.10': Connection refused > (Connection refused) > 2021-03-07 10:29:20.383 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > - Failed to connect from address '/0:0:0:0:0:0:0:1%lo': Network is > unreachable (connect failed) > 2021-03-07 10:29:20.383 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > - Failed to connect from address '/127.0.0.1': Connection refused > (Connection refused) > 2021-03-07 10:29:20.384 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > - Failed to connect from address '/fe80:0:0:0:20c:29ff:fea1:6d6b%ens33': > Network is unreachable (connect failed) > 2021-03-07 10:29:20.384 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > - Failed to connect from address '/192.168.31.10': Connection refused > (Connection refused) > 2021-03-07 10:29:20.384 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > - Failed to connect from address '/0:0:0:0:0:0:0:1%lo': Network is > unreachable (connect failed) > 2021-03-07 10:29:20.385 [main] INFO > org.apache.flink.runtime.net.ConnectionUtils > - Failed to connect from address '/127.0.0.1': Connection refused > (Connection refused) > 2021-03-07 10:29:20
【flink sql-client 读写 Kerberos认证的hive】
环境： flink1.12.1 hive2.1.0 CDH6.2.0 【问题描述】 在没开启Kerberos认证时，可以正常读写hive表 开启Kerberos认证后， 启动时可以正常读取到hive metastore的元数据信息，读写不了表。 【sql-client.sh embedded】 Flink SQL> show tables; dimension_table dimension_table1 test Flink SQL> select * from test; [ERROR] Could not execute SQL statement. Reason: org.apache.flink.connectors.hive.FlinkHiveException: Failed to collect all partitions from hive metaStore 【完整日志 /opt/cloudera/parcels/FLINK-1.12.1-BIN-SCALA_2.11/lib/flink/log/flink-root-sql-client-cdh6.com.log】 2021-03-07 10:29:18.776 [main] INFO org.apache.flink.runtime.net.ConnectionUtils - Trying to connect to localhost/127.0.0.1:6123 2021-03-07 10:29:18.777 [main] INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address 'cdh6.com/192.168.31.10': Connection refused (Connection refused) 2021-03-07 10:29:18.778 [main] INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/127.0.0.1': Connection refused (Connection refused) 2021-03-07 10:29:18.778 [main] INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/fe80:0:0:0:20c:29ff:fea1:6d6b%ens33': Network is unreachable (connect failed) 2021-03-07 10:29:18.778 [main] INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/192.168.31.10': Connection refused (Connection refused) 2021-03-07 10:29:18.779 [main] INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/0:0:0:0:0:0:0:1%lo': Network is unreachable (connect failed) 2021-03-07 10:29:18.779 [main] INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/127.0.0.1': Connection refused (Connection refused) 2021-03-07 10:29:18.779 [main] INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/fe80:0:0:0:20c:29ff:fea1:6d6b%ens33': Network is unreachable (connect failed) 2021-03-07 10:29:18.779 [main] INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/192.168.31.10': Connection refused (Connection refused) 2021-03-07 10:29:18.780 [main] INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/0:0:0:0:0:0:0:1%lo': Network is unreachable (connect failed) 2021-03-07 10:29:18.780 [main] INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/127.0.0.1': Connection refused (Connection refused) 2021-03-07 10:29:18.780 [main] INFO org.apache.flink.runtime.net.ConnectionUtils - Could not connect. Waiting for 1600 msecs before next attempt 2021-03-07 10:29:20.381 [main] INFO org.apache.flink.runtime.net.ConnectionUtils - Trying to connect to localhost/127.0.0.1:6123 2021-03-07 10:29:20.381 [main] INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address 'cdh6.com/192.168.31.10': Connection refused (Connection refused) 2021-03-07 10:29:20.382 [main] INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/127.0.0.1': Connection refused (Connection refused) 2021-03-07 10:29:20.383 [main] INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/fe80:0:0:0:20c:29ff:fea1:6d6b%ens33': Network is unreachable (connect failed) 2021-03-07 10:29:20.383 [main] INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/192.168.31.10': Connection refused (Connection refused) 2021-03-07 10:29:20.383 [main] INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/0:0:0:0:0:0:0:1%lo': Network is unreachable (connect failed) 2021-03-07 10:29:20.383 [main] INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/127.0.0.1': Connection refused (Connection refused) 2021-03-07 10:29:20.384 [main] INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/fe80:0:0:0:20c:29ff:fea1:6d6b%ens33': Network is unreachable (connect failed) 2021-03-07 10:29:20.384 [main] INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/192.168.31.10': Connection refused (Connection refused) 2021-03-07 10:29:20.384 [main] INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/0:0:0:0:0:0:0:1%lo': Network is unreachable (connect failed) 2021-03-07 10:29:20.385 [main] INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/127.0.0.1': Connection refused (Connection refused) 2021-03-07 10:29:20.385 [main] INFO org.apache.flink.runtime.net.ConnectionUtils - Could not connect. Waiting for 1829 msecs before next attempt 2021-03-07 10:29:22.214 [main] INFO org.apache.flink.runtime.net.ConnectionUtils - Trying to connect to localhost/127.0.0.1:6123 2021-03-07 10:29:22.215 [main] INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address 'cdh6.com/192.168.31.10': Connection refu
