[jira] [Commented] (HIVE-16529) Replace JPAM with libpam4j for PAM authentication
[
https://issues.apache.org/jira/browse/HIVE-16529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16139440#comment-16139440
]
Eric Yang commented on HIVE-16529:
--
JPAM user account expiration issue can easily work around by applying this
patch to JPAM:
{code}
--- jpam/jpam/src/c/Pam.c 2005-06-14 20:02:36.0 -0700
+++ ../../jpam/jpam/jpam/src/c/Pam.c2017-08-23 18:20:09.0 -0700
@@ -151,6 +151,9 @@
printf("***Sending password\n");
reply[replies].resp = COPY_STRING(password);
}
+ if (msg[replies]->msg_style==4) {
+ reply[replies].resp = NULL;
+ }
if (debug)
printf("***Response to PAM is: |%s|\n", reply[replies].resp);
}
{code}
This might be a workaround solution instead of replacing JPAM with libpam4j.
> Replace JPAM with libpam4j for PAM authentication
> -
>
> Key: HIVE-16529
> URL: https://issues.apache.org/jira/browse/HIVE-16529
> Project: Hive
> Issue Type: Improvement
> Components: Authentication
>Affects Versions: 1.2.0
>Reporter: Richard Ding
>Assignee: Sailaja Navvluru
>
> PAM authentication is an important feature available since Hive 0.13. But
> Hive blog gives the following warnings:
> {quote}
> JPAM library that is used to provide the PAM authentication mode can cause
> HiveServer2 to go down if a user's password has expired. This happens because
> of segfault/core dumps from native code invoked by JPAM. Some users have also
> reported crashes during logins in other cases as well. Use of LDAP or
> KERBEROS is recommended.
> {quote}
> JPAM also requires user to install a native library. Furthermore, JPAM
> library seems not to have been updated since 2007.
> Other Apache projects (e.g. Ambari/Ranger/Knox) use a newer library libpam4j
> which doesn't require installation of native library.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (HIVE-17126) Hive Metastore is incompatible with MariaDB 10.x
[ https://issues.apache.org/jira/browse/HIVE-17126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang resolved HIVE-17126. -- Resolution: Not A Problem I am incline to close this issue as user error for mismatching mysql-connector-java driver with mariadb versions. > Hive Metastore is incompatible with MariaDB 10.x > > > Key: HIVE-17126 > URL: https://issues.apache.org/jira/browse/HIVE-17126 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.0, 1.1.0, 2.0.0 >Reporter: Eric Yang > > MariaDB 10.x is commonly used for cheap RDBMS high availability. Hive usage > of Datanucleus is currently preventing Hive Metastore to use MariaDB 10.x as > highly available metastore. Datanucleus generate SQL statements that are not > parsable by MariaDB 10.x when dropping Hive table or database schema. > Without MariaDB HA setup, the SQL statement problem also exists for metastore > interaction with MariaDB 10.x. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17126) Hive Metastore is incompatible with MariaDB 10.x
[
https://issues.apache.org/jira/browse/HIVE-17126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119004#comment-16119004
]
Eric Yang commented on HIVE-17126:
--
[~sershe] SET OPTION is removed from MySQL 5.6 and newer. There is similar
error reported to [MariaDB
community|https://mariadb.atlassian.net/browse/MDEV-6201]. Perhaps, there is
something in the driver layer that prevented the session to be set for
ANSI_QUOTES. The system was using
{{mysql-connector-java-5.1.17-6.el6.noarch}}, which came with RHEL6.x family.
This might be problematic because the driver and server are not fully
compatible with each other. I will do more testing this weekend with a new
version of MariaDB connector/J to see if we can side step this issue. I think
it is important to do a error check for SET @@session.sql_mode with more user
friendly message. The current code seems to execute SQL queries even if the
SET query failed to execute.
> Hive Metastore is incompatible with MariaDB 10.x
>
>
> Key: HIVE-17126
> URL: https://issues.apache.org/jira/browse/HIVE-17126
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.2.0, 1.1.0, 2.0.0
>Reporter: Eric Yang
>
> MariaDB 10.x is commonly used for cheap RDBMS high availability. Hive usage
> of Datanucleus is currently preventing Hive Metastore to use MariaDB 10.x as
> highly available metastore. Datanucleus generate SQL statements that are not
> parsable by MariaDB 10.x when dropping Hive table or database schema.
> Without MariaDB HA setup, the SQL statement problem also exists for metastore
> interaction with MariaDB 10.x.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17126) Hive Metastore is incompatible with MariaDB 10.x
[ https://issues.apache.org/jira/browse/HIVE-17126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115511#comment-16115511 ] Eric Yang commented on HIVE-17126: -- >From the log output, it appears that SQL statement generated by DataNucleus is >double quoting table names and variable names. The syntax is invalid for >MariaDB. It appears that DataNucleus failed to detect dialect properly for >this specific combination of OS and MariaDB. I agree that HIVE-14870 can >solve this problem in the long run. If there is something that can be done in >detection of SQL dialect, maybe we can have a solution sooner for this issue. > Hive Metastore is incompatible with MariaDB 10.x > > > Key: HIVE-17126 > URL: https://issues.apache.org/jira/browse/HIVE-17126 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.0, 1.1.0, 2.0.0 >Reporter: Eric Yang > > MariaDB 10.x is commonly used for cheap RDBMS high availability. Hive usage > of Datanucleus is currently preventing Hive Metastore to use MariaDB 10.x as > highly available metastore. Datanucleus generate SQL statements that are not > parsable by MariaDB 10.x when dropping Hive table or database schema. > Without MariaDB HA setup, the SQL statement problem also exists for metastore > interaction with MariaDB 10.x. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17126) Hive Metastore is incompatible with MariaDB 10.x
[
https://issues.apache.org/jira/browse/HIVE-17126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115508#comment-16115508
]
Eric Yang commented on HIVE-17126:
--
Output from hive shell:
{code}
hive> drop table employee;
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Exception thrown
when executing query)
hive>
{code}
> Hive Metastore is incompatible with MariaDB 10.x
>
>
> Key: HIVE-17126
> URL: https://issues.apache.org/jira/browse/HIVE-17126
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.2.0, 1.1.0, 2.0.0
>Reporter: Eric Yang
>
> MariaDB 10.x is commonly used for cheap RDBMS high availability. Hive usage
> of Datanucleus is currently preventing Hive Metastore to use MariaDB 10.x as
> highly available metastore. Datanucleus generate SQL statements that are not
> parsable by MariaDB 10.x when dropping Hive table or database schema.
> Without MariaDB HA setup, the SQL statement problem also exists for metastore
> interaction with MariaDB 10.x.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17126) Hive Metastore is incompatible with MariaDB 10.x
[
https://issues.apache.org/jira/browse/HIVE-17126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115507#comment-16115507
]
Eric Yang commented on HIVE-17126:
--
Stack trace for the failure:
{code}
2017-08-05 13:42:25,984 ERROR bonecp.BoneCP
(BoneCP.java:destroyConnection(221)) - Error in attempting to close connection
java.sql.SQLException: Unknown system variable 'OPTION'
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2619)
at com.mysql.jdbc.ConnectionImpl.unsetMaxRows(ConnectionImpl.java:5421)
at com.mysql.jdbc.StatementImpl.realClose(StatementImpl.java:2441)
at
com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:3079)
at
com.mysql.jdbc.ConnectionImpl.closeAllOpenStatements(ConnectionImpl.java:1585)
at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4361)
at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1557)
at
com.jolbox.bonecp.ConnectionHandle.internalClose(ConnectionHandle.java:549)
at com.jolbox.bonecp.BoneCP.destroyConnection(BoneCP.java:219)
at
com.jolbox.bonecp.ConnectionHandle.markPossiblyBroken(ConnectionHandle.java:390)
at
com.jolbox.bonecp.PreparedStatementHandle.executeQuery(PreparedStatementHandle.java:183)
at
org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeQuery(ParamLoggingPreparedStatement.java:381)
at
org.datanucleus.store.rdbms.SQLController.executeStatementQuery(SQLController.java:504)
at
org.datanucleus.store.rdbms.query.SQLQuery.performExecute(SQLQuery.java:280)
at org.datanucleus.store.query.Query.executeQuery(Query.java:1786)
at
org.datanucleus.store.query.AbstractSQLQuery.executeWithArray(AbstractSQLQuery.java:339)
at org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:312)
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:1660)
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilterInternal(MetaStoreDirectSql.java:483)
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitions(MetaStoreDirectSql.java:403)
at
org.apache.hadoop.hive.metastore.ObjectStore$2.getSqlResult(ObjectStore.java:1735)
at
org.apache.hadoop.hive.metastore.ObjectStore$2.getSqlResult(ObjectStore.java:1731)
at
org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2391)
at
org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsInternal(ObjectStore.java:1742)
at
org.apache.hadoop.hive.metastore.ObjectStore.getPartitions(ObjectStore.java:1725)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
at com.sun.proxy.$Proxy2.getPartitions(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.dropPartitionsAndGetLocations(HiveMetaStore.java:1693)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:1532)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:1737)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
at com.sun.proxy.$Proxy4.drop_table_with_environment_context(Unknown
Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:9256)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:9240)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
at
[jira] [Commented] (HIVE-17126) Hive Metastore is incompatible with MariaDB 10.x
[ https://issues.apache.org/jira/browse/HIVE-17126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115506#comment-16115506 ] Eric Yang commented on HIVE-17126: -- This seems to affect RHEL/CentOS 6.8 with MariaDB 10.x release. It has been reported that CentOS 7.x works fine. > Hive Metastore is incompatible with MariaDB 10.x > > > Key: HIVE-17126 > URL: https://issues.apache.org/jira/browse/HIVE-17126 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.0, 1.1.0, 2.0.0 >Reporter: Eric Yang > > MariaDB 10.x is commonly used for cheap RDBMS high availability. Hive usage > of Datanucleus is currently preventing Hive Metastore to use MariaDB 10.x as > highly available metastore. Datanucleus generate SQL statements that are not > parsable by MariaDB 10.x when dropping Hive table or database schema. > Without MariaDB HA setup, the SQL statement problem also exists for metastore > interaction with MariaDB 10.x. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17187) WebHCat SPNEGO support is incompleted
[ https://issues.apache.org/jira/browse/HIVE-17187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103740#comment-16103740 ] Eric Yang commented on HIVE-17187: -- See [the blog|https://developer.ibm.com/hadoop/2016/05/12/hbase-rest-gateway-security/] written by IBM about SPNEGO for HBase REST API. This is a good source to implement SPNEGO properly with doAs calls with service principal instead of proxy user with SPNEGO credential. > WebHCat SPNEGO support is incompleted > - > > Key: HIVE-17187 > URL: https://issues.apache.org/jira/browse/HIVE-17187 > Project: Hive > Issue Type: Bug > Components: WebHCat >Affects Versions: 1.2.1 >Reporter: Eric Yang > > [Some online > document|https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_security/content/spnego_setup_for_webhcat.html] > describes how to setup WebHCat with SPNEGO support. However, there could be > multiple services use SPNEGO on the same host. For example, HBase REST API > can also setup to use HTTP principal for SPNEGO support. When HTTP principal > is shared among other services, Hadoop proxy user settings can not identify > the origin of doAs call with HTTP principal, is invoked by HBase REST API or > WebHCat. Ideally, WebHCat should keep track of its own service principal > independent of SPNEGO principal to ensure that SPNEGO principal is only given > authentication access. SPNEGO principal should not be used in proxy user > setting to grant authorization access. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
