Re: [SECURITY] CVE-2018-1314: Hive explain query not being authorized
Daniel - Is this happening when beeline security is enabled? Can you provide a link for more info on this? On Wed, Nov 7, 2018 at 14:25 Daniel Dai wrote: > CVE-2018-1314: Hive explain query not being authorized > > Severity: Important > > Vendor: The Apache Software Foundation > > Versions Affected: This vulnerability affects all versions of Hive, > including 2.3.3, 3.1.0 and earlier > > Description: Hive "EXPLAIN" operation does not check for necessary > authorization of involved entities in a query. An unauthorized user > can do "EXPLAIN" on arbitrary table or view and expose table metadata > and statistics. > > Mitigation: all Hive users shall upgrade to 2.3.4 or 3.1.1 or later >
[SECURITY] CVE-2018-11777: Blocking local resource access in HiveServer2
CVE-2018-11777: Blocking local resource access in HiveServer2 Severity: Important Vendor: The Apache Software Foundation Versions Affected: This vulnerability affects all versions of Hive, including 2.3.3, 3.1.0 and earlier Description: Local resources on HiveServer2 machines are not properly protected against malicious user if ranger, sentry or sql standard authorizer is not in use. Mitigation: It is recommended to upgrade to 2.3.4 or 3.1.1 or later if HiveServer2 is used, and ranger, sentry or sql standard authorizer is not in use. Admin needs to specify the following entries in hiveserver2-site.xml: hive.security.authorization.enabled true hive.security.authorization.manager org.apache.hadoop.hive.ql.security.authorization.plugin.fallback.FallbackHiveAuthorizerFactory FallbackHiveAuthorizerFactory will do the following to mitigate above mentioned threat: 1. Disallow local file location in sql statements except for admin 2. Allow "set" only selected whitelist parameters 3. Disallow dfs commands except for admin 4. Disallow "ADD JAR" statement 5. Disallow "COMPILE" statement 6. Disallow "TRANSFORM" statement Credit: This issue was discovered by Mithun Radhakrishnan of Oath Inc
[SECURITY] CVE-2018-1314: Hive explain query not being authorized
CVE-2018-1314: Hive explain query not being authorized Severity: Important Vendor: The Apache Software Foundation Versions Affected: This vulnerability affects all versions of Hive, including 2.3.3, 3.1.0 and earlier Description: Hive "EXPLAIN" operation does not check for necessary authorization of involved entities in a query. An unauthorized user can do "EXPLAIN" on arbitrary table or view and expose table metadata and statistics. Mitigation: all Hive users shall upgrade to 2.3.4 or 3.1.1 or later
[ANNOUNCE] Apache Hive 2.3.4 Released
The Apache Hive team is proud to announce the release of Apache Hive version 2.3.4. The Apache Hive (TM) data warehouse software facilitates querying and managing large datasets residing in distributed storage. Built on top of Apache Hadoop (TM), it provides, among others: * Tools to enable easy data extract/transform/load (ETL) * A mechanism to impose structure on a variety of data formats * Access to files stored either directly in Apache HDFS (TM) or in other data storage systems such as Apache HBase (TM) * Query execution via Apache Hadoop MapReduce, Apache Tez and Apache Spark frameworks. For Hive release details and downloads, please visit: https://hive.apache.org/downloads.html Hive 2.3.4 Release Notes are available here: https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12344319=Text=12310843 We would like to thank the many contributors who made this release possible. Regards, The Apache Hive Team
Re: Create external table with s3 location error
Thanks for the logs. Couple of things here, 1. Based on the logs,the HiveServer2 seem to be down. This is an issue if you use Hue/Beeline 2. If you are using Hive CLI, you can still test it, where you have to add the (fs.s3a.access.key,fs.s3.secret.key)s3 keys in hdfs advanced configuration snippet for core-site.xml and hdfs-site.xml via CM.Also for location it should be s3a:// not s3:// On Wed, Nov 7, 2018, 2:58 PM Garry Chen Hi Suresh, > > I am using Hive 1.1.0-cdh5.14.4 and hive server log as below. > > > > 2018-11-07 19:43:16,581 WARN [main]: server.HiveServer2 > (HiveServer2.java:startHiveServer2(581)) - Error starting HiveServer2 on > attempt 1, will retry in 6ms > > java.lang.RuntimeException: > org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.RuntimeException: > org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.RuntimeException: Unable to instantiate > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient > > at > org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:840) > > at > org.apache.hadoop.hive.ql.session.SessionState.getAuthorizationMode(SessionState.java:1686) > > at > org.apache.hadoop.hive.ql.session.SessionState.isAuthorizationModeV2(SessionState.java:1697) > > at > org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1745) > > at > org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:125) > > at org.apache.hive.service.cli.CLIService.init(CLIService.java:111) > > at > org.apache.hive.service.CompositeService.init(CompositeService.java:59) > > at > org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:125) > > at > org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:542) > > at > org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:89) > > at > org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:793) > > at > org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:666) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at org.apache.hadoop.util.RunJar.run(RunJar.java:226) > > at org.apache.hadoop.util.RunJar.main(RunJar.java:141) > > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.RuntimeException: > org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.RuntimeException: Unable to instantiate > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient > > at > org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:391) > > at > org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:817) > > ... 17 more > > Caused by: java.lang.RuntimeException: > org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.RuntimeException: Unable to instantiate > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient > > at > org.apache.hadoop.hive.ql.security.authorization.HiveAuthorizationProviderBase.setConf(HiveAuthorizationProviderBase.java:114) > > at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73) > > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) > > at > org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:388) > > ... 18 more > > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.RuntimeException: Unable to instantiate > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient > > at > org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:220) > > at org.apache.hadoop.hive.ql.metadata.Hive.(Hive.java:338) > > at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:299) > > at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:274) > > at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:256) > > at > org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider.init(DefaultHiveAuthorizationProvider.java:29) > > at > org.apache.hadoop.hive.ql.security.authorization.HiveAuthorizationProviderBase.setConf(HiveAuthorizationProviderBase.java:112) > > ... 21 more > > Caused by: java.lang.RuntimeException: Unable to instantiate > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient > > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1638) > > at >
Re: Create external table with s3 location error
Hi Suresh, I am using Hive 1.1.0-cdh5.14.4 and hive server log as below. 2018-11-07 19:43:16,581 WARN [main]: server.HiveServer2 (HiveServer2.java:startHiveServer2(581)) - Error starting HiveServer2 on attempt 1, will retry in 6ms java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:840) at org.apache.hadoop.hive.ql.session.SessionState.getAuthorizationMode(SessionState.java:1686) at org.apache.hadoop.hive.ql.session.SessionState.isAuthorizationModeV2(SessionState.java:1697) at org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1745) at org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:125) at org.apache.hive.service.cli.CLIService.init(CLIService.java:111) at org.apache.hive.service.CompositeService.init(CompositeService.java:59) at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:125) at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:542) at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:89) at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:793) at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:666) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:226) at org.apache.hadoop.util.RunJar.main(RunJar.java:141) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:391) at org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:817) ... 17 more Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.security.authorization.HiveAuthorizationProviderBase.setConf(HiveAuthorizationProviderBase.java:114) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:388) ... 18 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:220) at org.apache.hadoop.hive.ql.metadata.Hive.(Hive.java:338) at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:299) at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:274) at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:256) at org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider.init(DefaultHiveAuthorizationProvider.java:29) at org.apache.hadoop.hive.ql.security.authorization.HiveAuthorizationProviderBase.setConf(HiveAuthorizationProviderBase.java:112) ... 21 more Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1638) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:67) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:82) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3411) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3430) at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3655) at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:231) at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:215) ... 27 more Caused by: java.lang.reflect.InvocationTargetException at
Re: Create external table with s3 location error
Are you using EMR or Apache hadoop open source? Can you share your hive megastore logs? On Wed, Nov 7, 2018, 2:19 PM Garry Chen hi All, > > I am try to create a external table using s3 as location > but failed. I add my access key and security key in hive-site.xml and > reboot the server. Any suggestion? > > > > hive> create external table kv (key int, values string) location > 's3://cu-iclick/test'; > > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.DDLTask. > MetaException(message:java.lang.NullPointerException) > > > > Garry >
Create external table with s3 location error
hi All, I am try to create a external table using s3 as location but failed. I add my access key and security key in hive-site.xml and reboot the server. Any suggestion? hive> create external table kv (key int, values string) location 's3://cu-iclick/test'; FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.lang.NullPointerException) Garry
