Matthias Pohl created FLINK-26224: ------------------------------------- Summary: HAJobRunOnHadoopS3FileSystemITCase fails with UnsupportedFileSystemSchemeException Key: FLINK-26224 URL: https://issues.apache.org/jira/browse/FLINK-26224 Project: Flink Issue Type: Improvement Components: Connectors / FileSystem Affects Versions: 1.15.0 Reporter: Matthias Pohl
We're observing occassional errors in a Flink fork based on {{master}} in {{HAJobRunOnHadoopS3FileSystemITCase}} where the Hadoop S3 FileSystem wasn't loaded properly causing the {{UnsupportedFileSystemSchemeException}}: {code} Feb 16 19:02:52 [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 7.391 s <<< FAILURE! - in org.apache.flink.fs.s3hadoop.HAJobRunOnHadoopS3FileSystemITCase Feb 16 19:02:52 [ERROR] org.apache.flink.fs.s3hadoop.HAJobRunOnHadoopS3FileSystemITCase.testJobExecutionInHaMode Time elapsed: 1.89 s <<< ERROR! Feb 16 19:02:52 java.io.IOException: Could not create FileSystem for highly available storage path (s3://bucket-cqycdk/test-cluster/default) Feb 16 19:02:52 at org.apache.flink.runtime.blob.BlobUtils.createFileSystemBlobStore(BlobUtils.java:102) Feb 16 19:02:52 at org.apache.flink.runtime.blob.BlobUtils.createBlobStoreFromConfig(BlobUtils.java:86) Feb 16 19:02:52 at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createZooKeeperHaServices(HighAvailabilityServicesUtils.java:87) Feb 16 19:02:52 at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createAvailableOrEmbeddedServices(HighAvailabilityServicesUtils.java:70) Feb 16 19:02:52 at org.apache.flink.runtime.minicluster.MiniCluster.createHighAvailabilityServices(MiniCluster.java:549) Feb 16 19:02:52 at org.apache.flink.runtime.minicluster.MiniCluster.start(MiniCluster.java:397) Feb 16 19:02:52 at org.apache.flink.runtime.testutils.MiniClusterResource.startMiniCluster(MiniClusterResource.java:245) Feb 16 19:02:52 at org.apache.flink.runtime.testutils.MiniClusterResource.before(MiniClusterResource.java:105) Feb 16 19:02:52 at org.apache.flink.runtime.testutils.MiniClusterExtension.before(MiniClusterExtension.java:56) Feb 16 19:02:52 at org.apache.flink.core.testutils.EachCallbackWrapper.beforeEach(EachCallbackWrapper.java:45) Feb 16 19:02:52 at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeBeforeEachCallbacks$2(TestMethodTestDescriptor.java:163) Feb 16 19:02:52 at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeBeforeMethodsOrCallbacksUntilExceptionOccurs$6(TestMethodTestDescriptor.java:199) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) Feb 16 19:02:52 at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeBeforeMethodsOrCallbacksUntilExceptionOccurs(TestMethodTestDescriptor.java:199) Feb 16 19:02:52 at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeBeforeEachCallbacks(TestMethodTestDescriptor.java:162) Feb 16 19:02:52 at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:129) Feb 16 19:02:52 at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) Feb 16 19:02:52 at java.util.ArrayList.forEach(ArrayList.java:1259) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) Feb 16 19:02:52 at java.util.ArrayList.forEach(ArrayList.java:1259) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57) Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54) Feb 16 19:02:52 at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107) Feb 16 19:02:52 at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88) Feb 16 19:02:52 at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54) Feb 16 19:02:52 at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67) Feb 16 19:02:52 at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52) Feb 16 19:02:52 at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114) Feb 16 19:02:52 at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86) Feb 16 19:02:52 at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86) Feb 16 19:02:52 at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53) Feb 16 19:02:52 at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:188) Feb 16 19:02:52 at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:154) Feb 16 19:02:52 at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124) Feb 16 19:02:52 at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428) Feb 16 19:02:52 at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162) Feb 16 19:02:52 at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562) Feb 16 19:02:52 at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548) Feb 16 19:02:52 Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 's3'. The scheme is directly supported by Flink through the following plugin: flink-s3-fs-presto. Please ensure that each plugin resides within its own subfolder within the plugins directory. See https://ci.apache.org/projects/flink/flink-docs-stable/ops/plugins.html for more information. If you want to use a Hadoop file system for that scheme, please add the scheme to the configuration fs.allowed-fallback-filesystems. For a full list of supported file systems, please see https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/. Feb 16 19:02:52 at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:514) Feb 16 19:02:52 at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:408) Feb 16 19:02:52 at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274) Feb 16 19:02:52 at org.apache.flink.runtime.blob.BlobUtils.createFileSystemBlobStore(BlobUtils.java:99) Feb 16 19:02:52 ... 63 more {code} -- This message was sent by Atlassian Jira (v8.20.1#820001)