[
https://issues.apache.org/jira/browse/DRILL-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943319#comment-14943319
]
Jacques Nadeau commented on DRILL-3893:
---------------------------------------
We do cache hive Metadata for something like one minute. We should enhance this
cache to refresh on a metadata miss.
> Issue with Drill after Hive Alters the Table
> ---------------------------------------------
>
> Key: DRILL-3893
> URL: https://issues.apache.org/jira/browse/DRILL-3893
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Hive, Storage - Hive
> Affects Versions: 1.0.0, 1.1.0
> Environment: DEV
> Reporter: arnab chatterjee
>
> The application team reproduced this again on another partitioned table with
> existing data.
> Providing some more details. I have enabled the version mode for errors.
> Drill is unable to fetch the new column name that was introduced.This most
> likely to me seems to me that it’s still picking up the stale metadata of
> hive.
> if (!tableColumns.contains(columnName)) {
> if (partitionNames.contains(columnName)) {
> selectedPartitionNames.add(columnName);
> } else {
> throw new ExecutionSetupException(String.format("Column %s does
> not exist", columnName));
> }
> }
> select testdata from allocation_test;
> Error: SYSTEM ERROR: ExecutionSetupException: Column testdata does not exist
> Fragment 0:0
> [Error Id: be5cccba-97f6-4cc4-94e8-c11a4c53c8f4 on
> redlxd00006.nomura.com:31010]
> (org.apache.drill.common.exceptions.ExecutionSetupException) Failure while
> initializing HiveRecordReader: Column testdata does not exist
> org.apache.drill.exec.store.hive.HiveRecordReader.init():241
> org.apache.drill.exec.store.hive.HiveRecordReader.<init>():138
> org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():58
> org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():34
> org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():150
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren():173
> org.apache.drill.exec.physical.impl.ImplCreator.getRootExec():106
> org.apache.drill.exec.physical.impl.ImplCreator.getExec():81
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():235
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1142
> java.util.concurrent.ThreadPoolExecutor$Worker.run():617
> java.lang.Thread.run():745
> Caused By (org.apache.drill.common.exceptions.ExecutionSetupException)
> Column testdata does not exist
> org.apache.drill.exec.store.hive.HiveRecordReader.init():206
> org.apache.drill.exec.store.hive.HiveRecordReader.<init>():138
> org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():58
> org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():34
> org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():150
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren():173
> org.apache.drill.exec.physical.impl.ImplCreator.getRootExec():106
> org.apache.drill.exec.physical.impl.ImplCreator.getExec():81
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():235
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1142
> java.util.concurrent.ThreadPoolExecutor$Worker.run():617
> java.lang.Thread.run():745 (state=,code=0)
> #########################################################
> We have hit the issue again, this time the application team added a new
> column to the table and once again Drill does not identify it.
> Last time it was an update to an existing column name.
> Please note that this is a partitioned table with existing data.
> Does Drill Cache the Meta somewhere and hence it’s not getting reflected
> immediately ?
> DRILL CLI
> > select legaltickettext from parentorders_asha;
> Error: SYSTEM ERROR: ExecutionSetupException: Column legaltickettext
> does not exist
> Fragment 0:0
> [Error Id: 62086e22-1341-459e-87ce-430a24cc5119 on
> redlxd00006.nomura.com:31010] (state=,code=0)
> HIVE CLI
> hive> describe formatted parentorders_asha;
> OK
> # col_name data_type comment
> recordtype string
> clientacronym string
> securityid string
> secdesc string
> side string
> orderqty bigint
> pricetype string
> price double
> ordercapacity string
> exdestination string
> cashmargin string
> ordernotes string
> traderid string
> transacttime timestamp
> transacttimens bigint
> commonid string
> omssrc string
> cumqty bigint
> expirytype string
> orderid string
> parentid string
> tradingaccount string
> businessarea string
> traderlocation string
> clientflowtype string
> complianceid string
> targetstrategy string
> expirydatetime string
> listid string
> mergedto string
> legaltikcetref string
> legaltickettext string
> # Partition Information
> # col_name data_type comment
> tradedate date
> ordertradetype string
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
