[
https://issues.apache.org/jira/browse/HADOOP-16186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16796456#comment-16796456
]
Gabor Bota commented on HADOOP-16186:
-------------------------------------
We get an NPE at {{DynamoDBMetadataStore.java:653}} so at
{{dirPathMeta.getLastUpdated()}} which means dirPathMeta is null.
We get to that line, so the expression {{metas.isEmpty() && dirPathMeta ==
null}} evaulates to false, which means {{metas.isEmpty()}} should be false.
{{metas}} is the metadata for the files in that path whereas {{dirPathMeta}} is
a directory metadata got from ddb AFTER getting the listing in the directory
from ddb.
h3. Understanding what's the issue
So let's see when can {{metas.isEmpty()}} evaluated false (so there are items
in the directory), but at the same time {{dirPathMeta}} is null (so there is no
directory metadata for that path in dynamo):
* Something can delete the directory metadata between those two calls: between
getting the file metas for that path and getting the dir metadata. Race
condition? Maybe: In the teardown the directory is deleted, so the files can be
there (let's say in a corner case there's still 1 file left when the file metas
are queried) but when we call for the directory meta it's deleted so we get a
null.
* There's no directory marker in ddb, but the files are there. That would be
an implementation issue when we create directory markers.
h3. Proposed solution
Log if this happens with an error. Handle it with checking the file metas and
dir meta separately.
> NPE in ITestS3AFileSystemContract teardown in
> DynamoDBMetadataStore.lambda$listChildren
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-16186
> URL: https://issues.apache.org/jira/browse/HADOOP-16186
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.0
> Reporter: Steve Loughran
> Assignee: Gabor Bota
> Priority: Major
>
> Test run options. NPE in test teardown
> {code}
> -Dparallel-tests -DtestsThreadCount=6 -Ds3guard -Ddynamodb
> {code}
> If you look at the code, its *exactly* the place fixed in HADOOP-15827, a
> change which HADOOP-15947 reverted.
> There's clearly some codepath which can surface which is causing failures in
> some situations, and having multiple patches switching between the && and ||
> operators isn't going to to fix it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
