[
https://issues.apache.org/jira/browse/IMPALA-9071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955542#comment-16955542
]
Joe McDonnell commented on IMPALA-9071:
---------------------------------------
>From CreateTableAsSelectStmt.java
>([https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java#L217)|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java#L217]:
{code:java}
if (msTbl.getSd().getLocation() == null ||
msTbl.getSd().getLocation().isEmpty()) {
msTbl.getSd().setLocation(getPathForNewTable(db, msTbl));
}{code}
[https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java#L241]
{code:java}
private static String getPathForNewTable(FeDb db, Table msTbl) {
String dbLocation = db.getMetaStoreDb().getLocationUri();
return new Path(dbLocation, msTbl.getTableName().toLowerCase()).toString();
}
{code}
It assumes the new table is placed under the database's location, which is no
longer true.
> When metastore.warehouse.dir != metastore.warehouse.external.dir, Impala
> writes to the wrong location for external tables
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: IMPALA-9071
> URL: https://issues.apache.org/jira/browse/IMPALA-9071
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Affects Versions: Impala 3.4.0
> Reporter: Joe McDonnell
> Priority: Blocker
> Labels: broken-build
>
> Hive introduced a translation layer that can convert a normal table to an
> external table. When doing so without a specified location, the translated
> external table uses metastore.warehouse.external.dir as the location rather
> than metastore.warehouse.dir. Impala does not know about this distinction, so
> it writes to the location it thinks the table should be (under
> metastore.warehouse.dir). This means I can do the following:
> {noformat}
> [localhost:21000] joetest> select count(*) from functional.alltypes;
> Query: select count(*) from functional.alltypes
> Query submitted at: 2019-10-19 13:08:24 (Coordinator:
> http://joemcdonnell:25000)
> Query progress can be monitored at:
> http://joemcdonnell:25000/query_plan?query_id=68434b05e2badd50:a18a2e3000000000
> +----------+
> | count(*) |
> +----------+
> | 7300 |
> +----------+
> Fetched 1 row(s) in 0.14s
> [localhost:21000] joetest> create table testtable as select * from
> functional.alltypes;
> Query: create table testtable as select * from functional.alltypes
> Query submitted at: 2019-10-19 13:08:36 (Coordinator:
> http://joemcdonnell:25000)
> Query progress can be monitored at:
> http://joemcdonnell:25000/query_plan?query_id=794b92fb68f36ab0:910d036400000000
> +----------------------+
> | summary |
> +----------------------+
> | Inserted 7300 row(s) |
> +----------------------+
> Fetched 1 row(s) in 0.50s
> [localhost:21000] joetest> select count(*) from testtable;
> Query: select count(*) from testtable
> Query submitted at: 2019-10-19 13:08:43 (Coordinator:
> http://joemcdonnell:25000)
> Query progress can be monitored at:
> http://joemcdonnell:25000/query_plan?query_id=66423abf016e65af:8362460900000000
> +----------+
> | count(*) |
> +----------+
> | 0 |
> +----------+
> Fetched 1 row(s) in 0.13s
> {noformat}
> We inserted 7300 rows, but we can't select them back because they were
> written to the wrong location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
