[
https://issues.apache.org/jira/browse/IMPALA-11592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17609618#comment-17609618
]
Joe McDonnell commented on IMPALA-11592:
----------------------------------------
I'm turning this into a more generic JIRA, because we've seen this symptom in
various s3 tests.
> NullPointerException from s3a.Listing.createObjectListingIterator in various
> S3 tests
> -------------------------------------------------------------------------------------
>
> Key: IMPALA-11592
> URL: https://issues.apache.org/jira/browse/IMPALA-11592
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
> Labels: broken-build
> Attachments: catalogd.INFO.gz, impalad.INFO.gz,
> impalad_node1.INFO.gz, impalad_node2.INFO.gz
>
>
> custom_cluster.test_local_catalog.TestLocalCatalogRetries.test_fetch_metadata_retry
> fails in a S3 build:
> {noformat}
> custom_cluster/test_local_catalog.py:317: in test_fetch_metadata_retry
> seen = self._check_metadata_retries(queries)
> custom_cluster/test_local_catalog.py:293: in _check_metadata_retries
> assert failed_queries.empty(),\
> E AssertionError: Failed query count non zero: [('refresh
> functional.alltypes', 'ImpalaBeeswaxException:\n Query
> aborted:TableLoadingException: Refreshing file and block metadata for 24
> paths for table functional.alltypes: failed to load 1 paths. Check the
> catalog server log for more details.\n\n')]
> E assert <bound method Queue.empty of <Queue.Queue instance at
> 0x7fe3fb710128>>()
> E + where <bound method Queue.empty of <Queue.Queue instance at
> 0x7fe3fb710128>> = <Queue.Queue instance at 0x7fe3fb710128>.empty
> {noformat}
> Looking into the catalog server log, there is a NullPointerException:
> {noformat}
> E0916 21:15:27.469425 25508 ParallelFileMetadataLoader.java:171] Refreshing
> file and block metadata for 24 paths for table functional.alltypes
> encountered an error loading data for path
> s3a://impala-test-uswest2-2/test-warehouse/alltypes/year=2010/month=9
> Java exception follows:
> java.util.concurrent.ExecutionException: java.lang.NullPointerException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.impala.catalog.ParallelFileMetadataLoader.loadInternal(ParallelFileMetadataLoader.java:168)
> at
> org.apache.impala.catalog.ParallelFileMetadataLoader.load(ParallelFileMetadataLoader.java:120)
> at
> org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:781)
> at org.apache.impala.catalog.HdfsTable.access$100(HdfsTable.java:153)
> at
> org.apache.impala.catalog.HdfsTable$PartitionDeltaUpdater.apply(HdfsTable.java:1534)
> at
> org.apache.impala.catalog.HdfsTable.updatePartitionsFromHms(HdfsTable.java:1411)
> at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1254)
> at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1179)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.reloadTable(CatalogServiceCatalog.java:2551)
> at
> org.apache.impala.service.CatalogOpExecutor.execResetMetadata(CatalogOpExecutor.java:6158)
> at
> org.apache.impala.service.JniCatalog.resetMetadata(JniCatalog.java:287)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.fs.s3a.Listing$ObjectListingIterator.<init>(Listing.java:621)
> at
> org.apache.hadoop.fs.s3a.Listing.createObjectListingIterator(Listing.java:163)
> at
> org.apache.hadoop.fs.s3a.Listing.createFileStatusListingIterator(Listing.java:144)
> at
> org.apache.hadoop.fs.s3a.Listing.getListFilesAssumingDir(Listing.java:212)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerListFiles(S3AFileSystem.java:4790)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listFiles$37(S3AFileSystem.java:4732)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:543)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:524)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:445)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2363)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2382)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.listFiles(S3AFileSystem.java:4731)
> at
> org.apache.impala.common.FileSystemUtil.listFiles(FileSystemUtil.java:754)
> at
> org.apache.impala.common.FileSystemUtil.listStatus(FileSystemUtil.java:729)
> at
> org.apache.impala.catalog.FileMetadataLoader.load(FileMetadataLoader.java:190)
> at
> org.apache.impala.catalog.ParallelFileMetadataLoader.lambda$loadInternal$1(ParallelFileMetadataLoader.java:162)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> E0916 21:15:27.469880 25508 JniCatalog.java:295] Error in resetMetadata for
> REFRESH TABLE functional.alltypes issued by jenkins. Time spent: 140ms.
> I0916 21:15:27.470046 25508 jni-util.cc:288]
> org.apache.impala.catalog.TableLoadingException: Refreshing file and block
> metadata for 24 paths for table functional.alltypes: failed to load 1 paths.
> Check the catalog server log for more details.
> at
> org.apache.impala.catalog.ParallelFileMetadataLoader.loadInternal(ParallelFileMetadataLoader.java:185)
> at
> org.apache.impala.catalog.ParallelFileMetadataLoader.load(ParallelFileMetadataLoader.java:120)
> at
> org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:781)
> at org.apache.impala.catalog.HdfsTable.access$100(HdfsTable.java:153)
> at
> org.apache.impala.catalog.HdfsTable$PartitionDeltaUpdater.apply(HdfsTable.java:1534)
> at
> org.apache.impala.catalog.HdfsTable.updatePartitionsFromHms(HdfsTable.java:1411)
> at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1254)
> at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1179)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.reloadTable(CatalogServiceCatalog.java:2551)
> at
> org.apache.impala.service.CatalogOpExecutor.execResetMetadata(CatalogOpExecutor.java:6158)
> at
> org.apache.impala.service.JniCatalog.resetMetadata(JniCatalog.java:287)
> {noformat}
> Dependent hadoop version: IMPALA_HADOOP_VERSION=3.1.1.7.2.16.0-164
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
