[ https://issues.apache.org/jira/browse/HADOOP-12999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Waldemar Hummer resolved HADOOP-12999. -------------------------------------- Resolution: Invalid > NPE when accessing (meta-)data via Hive query from S3 bucket > ------------------------------------------------------------ > > Key: HADOOP-12999 > URL: https://issues.apache.org/jira/browse/HADOOP-12999 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 > Affects Versions: 2.8.0, 2.7.2 > Environment: JDK8, Hive 2.0.0, Hadoop 2.7.2, also happens with Hadoop > 2.8.0-SNAPSHOT (git revision ab67b50543e2e9dc48f2dcc00de18c2e2c6b4647) > Reporter: Waldemar Hummer > > Querying data stored in S3 via Hive 2.0.0 causes a NPE. The exception occurs > when Hive Metastore uses hadoop-aws tools to query the bucket structure in S3. > Example Hive query: > {code} > create external table if not exists test_table_2 ( > id STRING, name STRING > ) > LOCATION 's3://my-bucket/test/test_insert2/'; > {code} > The required bucket folder exists properly in S3. (Also, there is a $folder$ > entry on the same directory level which is an often used workaround for S3 > tools that cannot handle empty folders): > {code} > $ s3cmd ls s3://my-bucket/test/test_insert2 > DIR s3://my-bucket/test/test_insert2/ > 2016-04-04 10:44 0 s3://my-bucket/test/test_insert2_$folder$ > {code} > The following is an excerpt of the stack trace in Hive console log: > {code} > exec.DDLTask (DDLTask.java:failed(541)) - > org.apache.hadoop.hive.ql.metadata.HiveException: > MetaException(message:java.lang.NullPointerException) > at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:783) > at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4032) > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:322) > ... > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.fs.s3.S3FileSystem.makeAbsolute(S3FileSystem.java:132) > at > org.apache.hadoop.fs.s3.S3FileSystem.getFileStatus(S3FileSystem.java:342) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426) > at org.apache.hadoop.hive.common.FileUtils.mkdir(FileUtils.java:518) > at org.apache.hadoop.hive.metastore.Warehouse.mkdirs(Warehouse.java:201) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1317) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1369) > ... 39 more > ``` > {code} > After digging into the code, it appears that the root cause for this issue is > not in Hive, but in the way that hadoop-aws is querying the bucket > information from S3. We have verified that the Path parameter passed into > org.apache.hadoop.hive.common.FileUtils.mkdir(...) is indeed NOT null. > The issue can be easily reproduced using the following standalone piece of > code: > {code} > FileSystem fs = new org.apache.hadoop.fs.s3.S3FileSystem(); > Configuration conf = new Configuration(); > conf.set("fs.s3.awsAccessKeyId", "..."); > conf.set("fs.s3.awsSecretAccessKey", "..."); > String url = "s3://my-bucket/test/test_insert2/"; > fs.initialize(new URI(url), conf); > Path f1 = new Path(url); > boolean inheritPerms = true; > org.apache.hadoop.hive.common.FileUtils.mkdir(fs, f1, inheritPerms, conf); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)