[ https://issues.apache.org/jira/browse/HADOOP-15796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16635524#comment-16635524 ]
Gabor Bota commented on HADOOP-15796: ------------------------------------- The following error keeps popping up when running tests ({{mvn clean verify -Dscale}}) without s3guard and no parallel tests. {{noformat}} [INFO] Running org.apache.hadoop.fs.s3a.commit.staging.integration.ITDirectoryCommitMRJob [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.963 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.commit.staging.integration.ITDirectoryCommitMRJob [ERROR] testMRJob(org.apache.hadoop.fs.s3a.commit.staging.integration.ITDirectoryCommitMRJob) Time elapsed: 0.029 s <<< ERROR! java.io.IOException: s3a://cloudera-dev-gabor-ireland: FileSystem is closed! [INFO] Running org.apache.hadoop.fs.s3a.commit.staging.integration.ITStagingCommitMRJob [ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 3.771 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.commit.staging.integration.ITStagingCommitMRJob [ERROR] testStagingDirectory(org.apache.hadoop.fs.s3a.commit.staging.integration.ITStagingCommitMRJob) Time elapsed: 0.022 s <<< ERROR! java.io.IOException: s3a://cloudera-dev-gabor-ireland: FileSystem is closed! [ERROR] testMRJob(org.apache.hadoop.fs.s3a.commit.staging.integration.ITStagingCommitMRJob) Time elapsed: 0.019 s <<< ERROR! java.io.IOException: s3a://cloudera-dev-gabor-ireland: FileSystem is closed! [INFO] Running org.apache.hadoop.fs.s3a.commit.staging.integration.ITStagingCommitMRJobBadDest [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.771 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.commit.staging.integration.ITStagingCommitMRJobBadDest [ERROR] testMRJob(org.apache.hadoop.fs.s3a.commit.staging.integration.ITStagingCommitMRJobBadDest) Time elapsed: 0.02 s <<< ERROR! java.io.IOException: s3a://cloudera-dev-gabor-ireland: FileSystem is closed! [INFO] Running org.apache.hadoop.fs.s3a.commit.magic.ITMagicCommitMRJob [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.978 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.commit.magic.ITMagicCommitMRJob [ERROR] testMRJob(org.apache.hadoop.fs.s3a.commit.magic.ITMagicCommitMRJob) Time elapsed: 0.021 s <<< ERROR! java.io.IOException: s3a://cloudera-dev-gabor-ireland: FileSystem is closed! {{noformat}} We get a lot of these errors downstream as well. Maybe I should create a separate jira to fix this. I think there could be an error how we handle fs caching. > ClassCastException: S3AFileSystem cannot be cast to MockS3AFileSystem when fs > caching is disabled > ------------------------------------------------------------------------------------------------- > > Key: HADOOP-15796 > URL: https://issues.apache.org/jira/browse/HADOOP-15796 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Affects Versions: 3.1.1 > Reporter: Gabor Bota > Priority: Major > > Sometimes we get {{java.io.IOException: s3a://TEST-BUCKET: FileSystem is > closed!}} when running tests, because of filesystem caching: an fs instance > is closed from a test teardown while another test is still using it. (That > should be fixed in another issue.) > To avoid test errors introduced by closed fs I've disabled fs caching while > testing. This caused the following issue in lots of test: > {noformat} > [ERROR] > testPartitionsResolution(org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedFileListing) > Time elapsed: 1.994 s <<< ERROR! > java.lang.ClassCastException: org.apache.hadoop.fs.s3a.S3AFileSystem cannot > be cast to org.apache.hadoop.fs.s3a.MockS3AFileSystem > at > org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase.lookupWrapperFS(StagingTestBase.java:145) > at > org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase$JobCommitterTest.setupJob(StagingTestBase.java:294) > {noformat} > We should fix the casting during these tests. > To switch of fs caching during a test (mvn verify) run, add the following to > the config: > {code:java} > <property> > <name>fs.s3a.impl.disable.cache</name> > <value>true</value> > </property> > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org