[jira] [Commented] (HIVE-16529) Replace JPAM with libpam4j for PAM authentication
[ https://issues.apache.org/jira/browse/HIVE-16529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16139440#comment-16139440 ] Eric Yang commented on HIVE-16529: -- JPAM user account expiration issue can easily work around by applying this patch to JPAM: {code} --- jpam/jpam/src/c/Pam.c 2005-06-14 20:02:36.0 -0700 +++ ../../jpam/jpam/jpam/src/c/Pam.c2017-08-23 18:20:09.0 -0700 @@ -151,6 +151,9 @@ printf("***Sending password\n"); reply[replies].resp = COPY_STRING(password); } + if (msg[replies]->msg_style==4) { + reply[replies].resp = NULL; + } if (debug) printf("***Response to PAM is: |%s|\n", reply[replies].resp); } {code} This might be a workaround solution instead of replacing JPAM with libpam4j. > Replace JPAM with libpam4j for PAM authentication > - > > Key: HIVE-16529 > URL: https://issues.apache.org/jira/browse/HIVE-16529 > Project: Hive > Issue Type: Improvement > Components: Authentication >Affects Versions: 1.2.0 >Reporter: Richard Ding >Assignee: Sailaja Navvluru > > PAM authentication is an important feature available since Hive 0.13. But > Hive blog gives the following warnings: > {quote} > JPAM library that is used to provide the PAM authentication mode can cause > HiveServer2 to go down if a user's password has expired. This happens because > of segfault/core dumps from native code invoked by JPAM. Some users have also > reported crashes during logins in other cases as well. Use of LDAP or > KERBEROS is recommended. > {quote} > JPAM also requires user to install a native library. Furthermore, JPAM > library seems not to have been updated since 2007. > Other Apache projects (e.g. Ambari/Ranger/Knox) use a newer library libpam4j > which doesn't require installation of native library. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HIVE-17126) Hive Metastore is incompatible with MariaDB 10.x
[ https://issues.apache.org/jira/browse/HIVE-17126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang resolved HIVE-17126. -- Resolution: Not A Problem I am incline to close this issue as user error for mismatching mysql-connector-java driver with mariadb versions. > Hive Metastore is incompatible with MariaDB 10.x > > > Key: HIVE-17126 > URL: https://issues.apache.org/jira/browse/HIVE-17126 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.0, 1.1.0, 2.0.0 >Reporter: Eric Yang > > MariaDB 10.x is commonly used for cheap RDBMS high availability. Hive usage > of Datanucleus is currently preventing Hive Metastore to use MariaDB 10.x as > highly available metastore. Datanucleus generate SQL statements that are not > parsable by MariaDB 10.x when dropping Hive table or database schema. > Without MariaDB HA setup, the SQL statement problem also exists for metastore > interaction with MariaDB 10.x. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17126) Hive Metastore is incompatible with MariaDB 10.x
[ https://issues.apache.org/jira/browse/HIVE-17126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119004#comment-16119004 ] Eric Yang commented on HIVE-17126: -- [~sershe] SET OPTION is removed from MySQL 5.6 and newer. There is similar error reported to [MariaDB community|https://mariadb.atlassian.net/browse/MDEV-6201]. Perhaps, there is something in the driver layer that prevented the session to be set for ANSI_QUOTES. The system was using {{mysql-connector-java-5.1.17-6.el6.noarch}}, which came with RHEL6.x family. This might be problematic because the driver and server are not fully compatible with each other. I will do more testing this weekend with a new version of MariaDB connector/J to see if we can side step this issue. I think it is important to do a error check for SET @@session.sql_mode with more user friendly message. The current code seems to execute SQL queries even if the SET query failed to execute. > Hive Metastore is incompatible with MariaDB 10.x > > > Key: HIVE-17126 > URL: https://issues.apache.org/jira/browse/HIVE-17126 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.0, 1.1.0, 2.0.0 >Reporter: Eric Yang > > MariaDB 10.x is commonly used for cheap RDBMS high availability. Hive usage > of Datanucleus is currently preventing Hive Metastore to use MariaDB 10.x as > highly available metastore. Datanucleus generate SQL statements that are not > parsable by MariaDB 10.x when dropping Hive table or database schema. > Without MariaDB HA setup, the SQL statement problem also exists for metastore > interaction with MariaDB 10.x. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17126) Hive Metastore is incompatible with MariaDB 10.x
[ https://issues.apache.org/jira/browse/HIVE-17126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16115511#comment-16115511 ] Eric Yang commented on HIVE-17126: -- >From the log output, it appears that SQL statement generated by DataNucleus is >double quoting table names and variable names. The syntax is invalid for >MariaDB. It appears that DataNucleus failed to detect dialect properly for >this specific combination of OS and MariaDB. I agree that HIVE-14870 can >solve this problem in the long run. If there is something that can be done in >detection of SQL dialect, maybe we can have a solution sooner for this issue. > Hive Metastore is incompatible with MariaDB 10.x > > > Key: HIVE-17126 > URL: https://issues.apache.org/jira/browse/HIVE-17126 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.0, 1.1.0, 2.0.0 >Reporter: Eric Yang > > MariaDB 10.x is commonly used for cheap RDBMS high availability. Hive usage > of Datanucleus is currently preventing Hive Metastore to use MariaDB 10.x as > highly available metastore. Datanucleus generate SQL statements that are not > parsable by MariaDB 10.x when dropping Hive table or database schema. > Without MariaDB HA setup, the SQL statement problem also exists for metastore > interaction with MariaDB 10.x. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17126) Hive Metastore is incompatible with MariaDB 10.x
[ https://issues.apache.org/jira/browse/HIVE-17126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16115508#comment-16115508 ] Eric Yang commented on HIVE-17126: -- Output from hive shell: {code} hive> drop table employee; FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Exception thrown when executing query) hive> {code} > Hive Metastore is incompatible with MariaDB 10.x > > > Key: HIVE-17126 > URL: https://issues.apache.org/jira/browse/HIVE-17126 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.0, 1.1.0, 2.0.0 >Reporter: Eric Yang > > MariaDB 10.x is commonly used for cheap RDBMS high availability. Hive usage > of Datanucleus is currently preventing Hive Metastore to use MariaDB 10.x as > highly available metastore. Datanucleus generate SQL statements that are not > parsable by MariaDB 10.x when dropping Hive table or database schema. > Without MariaDB HA setup, the SQL statement problem also exists for metastore > interaction with MariaDB 10.x. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17126) Hive Metastore is incompatible with MariaDB 10.x
[ https://issues.apache.org/jira/browse/HIVE-17126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16115507#comment-16115507 ] Eric Yang commented on HIVE-17126: -- Stack trace for the failure: {code} 2017-08-05 13:42:25,984 ERROR bonecp.BoneCP (BoneCP.java:destroyConnection(221)) - Error in attempting to close connection java.sql.SQLException: Unknown system variable 'OPTION' at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2619) at com.mysql.jdbc.ConnectionImpl.unsetMaxRows(ConnectionImpl.java:5421) at com.mysql.jdbc.StatementImpl.realClose(StatementImpl.java:2441) at com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:3079) at com.mysql.jdbc.ConnectionImpl.closeAllOpenStatements(ConnectionImpl.java:1585) at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4361) at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1557) at com.jolbox.bonecp.ConnectionHandle.internalClose(ConnectionHandle.java:549) at com.jolbox.bonecp.BoneCP.destroyConnection(BoneCP.java:219) at com.jolbox.bonecp.ConnectionHandle.markPossiblyBroken(ConnectionHandle.java:390) at com.jolbox.bonecp.PreparedStatementHandle.executeQuery(PreparedStatementHandle.java:183) at org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeQuery(ParamLoggingPreparedStatement.java:381) at org.datanucleus.store.rdbms.SQLController.executeStatementQuery(SQLController.java:504) at org.datanucleus.store.rdbms.query.SQLQuery.performExecute(SQLQuery.java:280) at org.datanucleus.store.query.Query.executeQuery(Query.java:1786) at org.datanucleus.store.query.AbstractSQLQuery.executeWithArray(AbstractSQLQuery.java:339) at org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:312) at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:1660) at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilterInternal(MetaStoreDirectSql.java:483) at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitions(MetaStoreDirectSql.java:403) at org.apache.hadoop.hive.metastore.ObjectStore$2.getSqlResult(ObjectStore.java:1735) at org.apache.hadoop.hive.metastore.ObjectStore$2.getSqlResult(ObjectStore.java:1731) at org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2391) at org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsInternal(ObjectStore.java:1742) at org.apache.hadoop.hive.metastore.ObjectStore.getPartitions(ObjectStore.java:1725) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114) at com.sun.proxy.$Proxy2.getPartitions(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.dropPartitionsAndGetLocations(HiveMetaStore.java:1693) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:1532) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:1737) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107) at com.sun.proxy.$Proxy4.drop_table_with_environment_context(Unknown Source) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:9256) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:9240) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110) at org.apache.hadoop.hive.metastore.TUGIBasedP
[jira] [Commented] (HIVE-17126) Hive Metastore is incompatible with MariaDB 10.x
[ https://issues.apache.org/jira/browse/HIVE-17126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16115506#comment-16115506 ] Eric Yang commented on HIVE-17126: -- This seems to affect RHEL/CentOS 6.8 with MariaDB 10.x release. It has been reported that CentOS 7.x works fine. > Hive Metastore is incompatible with MariaDB 10.x > > > Key: HIVE-17126 > URL: https://issues.apache.org/jira/browse/HIVE-17126 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.0, 1.1.0, 2.0.0 >Reporter: Eric Yang > > MariaDB 10.x is commonly used for cheap RDBMS high availability. Hive usage > of Datanucleus is currently preventing Hive Metastore to use MariaDB 10.x as > highly available metastore. Datanucleus generate SQL statements that are not > parsable by MariaDB 10.x when dropping Hive table or database schema. > Without MariaDB HA setup, the SQL statement problem also exists for metastore > interaction with MariaDB 10.x. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17187) WebHCat SPNEGO support is incompleted
[ https://issues.apache.org/jira/browse/HIVE-17187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16103740#comment-16103740 ] Eric Yang commented on HIVE-17187: -- See [the blog|https://developer.ibm.com/hadoop/2016/05/12/hbase-rest-gateway-security/] written by IBM about SPNEGO for HBase REST API. This is a good source to implement SPNEGO properly with doAs calls with service principal instead of proxy user with SPNEGO credential. > WebHCat SPNEGO support is incompleted > - > > Key: HIVE-17187 > URL: https://issues.apache.org/jira/browse/HIVE-17187 > Project: Hive > Issue Type: Bug > Components: WebHCat >Affects Versions: 1.2.1 >Reporter: Eric Yang > > [Some online > document|https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_security/content/spnego_setup_for_webhcat.html] > describes how to setup WebHCat with SPNEGO support. However, there could be > multiple services use SPNEGO on the same host. For example, HBase REST API > can also setup to use HTTP principal for SPNEGO support. When HTTP principal > is shared among other services, Hadoop proxy user settings can not identify > the origin of doAs call with HTTP principal, is invoked by HBase REST API or > WebHCat. Ideally, WebHCat should keep track of its own service principal > independent of SPNEGO principal to ensure that SPNEGO principal is only given > authentication access. SPNEGO principal should not be used in proxy user > setting to grant authorization access. -- This message was sent by Atlassian JIRA (v6.4.14#64029)