[
https://issues.apache.org/jira/browse/IMPALA-9122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16982775#comment-16982775
]
Vihang Karajgaonkar commented on IMPALA-9122:
---------------------------------------------
Took a look at the logs. The test fails because event processor goes into a
error state after receiving this exception from the FileMetadataLoader
{noformat}
E1126 01:59:37.307276 46863 ParallelFileMetadataLoader.java:102] Refreshing
file and block metadata for 1 paths for table test_db.tbl_insert_part
encountered an error loading data for path
hdfs://localhost:20500/test-warehouse/test_db.db/tbl_insert_part/day=28/month=3/year=2019
Java exception follows:
java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File
hdfs://localhost:20500/test-warehouse/test_db.db/tbl_insert_part/day=28/month=3/year=2019/.hive-staging_hive_2019-11-26_01-59-35_788_4342075643583311236-2
does not exist.
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.impala.catalog.ParallelFileMetadataLoader.load(ParallelFileMetadataLoader.java:99)
at
org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:653)
at org.apache.impala.catalog.HdfsTable.reloadPartition(HdfsTable.java:2014)
at
org.apache.impala.catalog.CatalogServiceCatalog.reloadPartition(CatalogServiceCatalog.java:2592)
at
org.apache.impala.catalog.CatalogServiceCatalog.reloadPartitionIfExists(CatalogServiceCatalog.java:2225)
at
org.apache.impala.catalog.events.MetastoreEvents$InsertEvent.processPartitionInserts(MetastoreEvents.java:818)
at
org.apache.impala.catalog.events.MetastoreEvents$InsertEvent.process(MetastoreEvents.java:800)
at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:330)
at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:609)
at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:511)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: File
hdfs://localhost:20500/test-warehouse/test_db.db/tbl_insert_part/day=28/month=3/year=2019/.hive-staging_hive_2019-11-26_01-59-35_788_4342075643583311236-2
does not exist.
at
org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.<init>(DistributedFileSystem.java:1149)
at
org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.<init>(DistributedFileSystem.java:1156)
at
org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.<init>(DistributedFileSystem.java:1127)
at
org.apache.hadoop.hdfs.DistributedFileSystem$26.doCall(DistributedFileSystem.java:1106)
at
org.apache.hadoop.hdfs.DistributedFileSystem$26.doCall(DistributedFileSystem.java:1102)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusIterator(DistributedFileSystem.java:1114)
at
org.apache.impala.common.FileSystemUtil$RecursingIterator.handleFileStat(FileSystemUtil.java:751)
at
org.apache.impala.common.FileSystemUtil$RecursingIterator.hasNext(FileSystemUtil.java:724)
at
org.apache.impala.common.FileSystemUtil$FilterIterator.hasNext(FileSystemUtil.java:679)
at
org.apache.impala.catalog.FileMetadataLoader.load(FileMetadataLoader.java:176)
at
org.apache.impala.catalog.ParallelFileMetadataLoader.lambda$load$0(ParallelFileMetadataLoader.java:93)
at
org.apache.impala.catalog.ParallelFileMetadataLoader.lambda$load$0(ParallelFileMetadataLoader.java:93)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293)
at
com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:61)
at
com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:45)
at
org.apache.impala.catalog.ParallelFileMetadataLoader.load(ParallelFileMetadataLoader.java:93)
... 16 more
E1126 01:59:37.308281 46863 MetastoreEventsProcessor.java:519] Event processing
needs a invalidate command to resolve the state
Java exception follows:
org.apache.impala.catalog.events.MetastoreNotificationNeedsInvalidateException:
EventId: 19303 EventType: INSERT Refresh partition on table {} partition {}
failed. Event processing cannot continue. Issue and invalidate command to reset
the event processor state.
at
org.apache.impala.catalog.events.MetastoreEvents$InsertEvent.processPartitionInserts(MetastoreEvents.java:834)
at
org.apache.impala.catalog.events.MetastoreEvents$InsertEvent.process(MetastoreEvents.java:800)
at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:330)
at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:609)
at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:511)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.impala.catalog.TableLoadingException: Refreshing file and
block metadata for 1 paths for table test_db.tbl_insert_part: failed to load 1
paths. Check the catalog server log for more details.
at
org.apache.impala.catalog.ParallelFileMetadataLoader.load(ParallelFileMetadataLoader.java:116)
at
org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:653)
at org.apache.impala.catalog.HdfsTable.reloadPartition(HdfsTable.java:2014)
at
org.apache.impala.catalog.CatalogServiceCatalog.reloadPartition(CatalogServiceCatalog.java:2592)
at
org.apache.impala.catalog.CatalogServiceCatalog.reloadPartitionIfExists(CatalogServiceCatalog.java:2225)
at
org.apache.impala.catalog.events.MetastoreEvents$InsertEvent.processPartitionInserts(MetastoreEvents.java:818)
... 11 more
{noformat}
As the trace shows above the exception comes from
{{org.apache.hadoop.hdfs.DistributedFileSystem.listStatusIterator}} hive
deletes the intermediate directories after the query completes while the
catalog is trying to load the directories.
I think in such cases it is safe to ignore these exceptions in catalog and let
the loading continue.
> TestEventProcessing.test_insert_events flaky in precommit
> ---------------------------------------------------------
>
> Key: IMPALA-9122
> URL: https://issues.apache.org/jira/browse/IMPALA-9122
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Reporter: Tim Armstrong
> Assignee: Anurag Mantripragada
> Priority: Critical
> Labels: broken-build, flaky
>
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/8613/
> {noformat}
> custom_cluster.test_event_processing.TestEventProcessing.test_insert_events
> (from pytest)
> Failing for the past 1 build (Since Failed#8613 )
> Took 1 min 20 sec.
> add description
> Error Message
> assert <bound method TestEventProcessing.wait_for_insert_event_processing of
> <test_event_processing.TestEventProcessing object at 0x7f7fa86ec6d0>>(18421)
> is True + where <bound method
> TestEventProcessing.wait_for_insert_event_processing of
> <test_event_processing.TestEventProcessing object at 0x7f7fa86ec6d0>> =
> <test_event_processing.TestEventProcessing object at
> 0x7f7fa86ec6d0>.wait_for_insert_event_processing
> Stacktrace
> custom_cluster/test_event_processing.py:82: in test_insert_events
> self.run_test_insert_events()
> custom_cluster/test_event_processing.py:143: in run_test_insert_events
> assert self.wait_for_insert_event_processing(last_synced_event_id) is True
> E assert <bound method TestEventProcessing.wait_for_insert_event_processing
> of <test_event_processing.TestEventProcessing object at
> 0x7f7fa86ec6d0>>(18421) is True
> E + where <bound method
> TestEventProcessing.wait_for_insert_event_processing of
> <test_event_processing.TestEventProcessing object at 0x7f7fa86ec6d0>> =
> <test_event_processing.TestEventProcessing object at
> 0x7f7fa86ec6d0>.wait_for_insert_event_processing
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
