[ https://issues.apache.org/jira/browse/IMPALA-9071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955542#comment-16955542 ]
Joe McDonnell commented on IMPALA-9071: --------------------------------------- >From CreateTableAsSelectStmt.java >([https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java#L217)|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java#L217]: {code:java} if (msTbl.getSd().getLocation() == null || msTbl.getSd().getLocation().isEmpty()) { msTbl.getSd().setLocation(getPathForNewTable(db, msTbl)); }{code} [https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java#L241] {code:java} private static String getPathForNewTable(FeDb db, Table msTbl) { String dbLocation = db.getMetaStoreDb().getLocationUri(); return new Path(dbLocation, msTbl.getTableName().toLowerCase()).toString(); } {code} It assumes the new table is placed under the database's location, which is no longer true. > When metastore.warehouse.dir != metastore.warehouse.external.dir, Impala > writes to the wrong location for external tables > ------------------------------------------------------------------------------------------------------------------------- > > Key: IMPALA-9071 > URL: https://issues.apache.org/jira/browse/IMPALA-9071 > Project: IMPALA > Issue Type: Bug > Components: Frontend > Affects Versions: Impala 3.4.0 > Reporter: Joe McDonnell > Priority: Blocker > Labels: broken-build > > Hive introduced a translation layer that can convert a normal table to an > external table. When doing so without a specified location, the translated > external table uses metastore.warehouse.external.dir as the location rather > than metastore.warehouse.dir. Impala does not know about this distinction, so > it writes to the location it thinks the table should be (under > metastore.warehouse.dir). This means I can do the following: > {noformat} > [localhost:21000] joetest> select count(*) from functional.alltypes; > Query: select count(*) from functional.alltypes > Query submitted at: 2019-10-19 13:08:24 (Coordinator: > http://joemcdonnell:25000) > Query progress can be monitored at: > http://joemcdonnell:25000/query_plan?query_id=68434b05e2badd50:a18a2e3000000000 > +----------+ > | count(*) | > +----------+ > | 7300 | > +----------+ > Fetched 1 row(s) in 0.14s > [localhost:21000] joetest> create table testtable as select * from > functional.alltypes; > Query: create table testtable as select * from functional.alltypes > Query submitted at: 2019-10-19 13:08:36 (Coordinator: > http://joemcdonnell:25000) > Query progress can be monitored at: > http://joemcdonnell:25000/query_plan?query_id=794b92fb68f36ab0:910d036400000000 > +----------------------+ > | summary | > +----------------------+ > | Inserted 7300 row(s) | > +----------------------+ > Fetched 1 row(s) in 0.50s > [localhost:21000] joetest> select count(*) from testtable; > Query: select count(*) from testtable > Query submitted at: 2019-10-19 13:08:43 (Coordinator: > http://joemcdonnell:25000) > Query progress can be monitored at: > http://joemcdonnell:25000/query_plan?query_id=66423abf016e65af:8362460900000000 > +----------+ > | count(*) | > +----------+ > | 0 | > +----------+ > Fetched 1 row(s) in 0.13s > {noformat} > We inserted 7300 rows, but we can't select them back because they were > written to the wrong location. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org