Hi, Gabar, Thanks to reply, I make sure that not conflict in maven, all the hadoop dependency is in provided scope, and checked my result jar it not contains (src/main/resources/META-INF/services).
This is my pom: <project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> <modelVersion>4.0.0</modelVersion> <groupId>com.test.flink</groupId> <artifactId>flink-sync</artifactId> <version>1.0-SNAPSHOT</version> <packaging>jar</packaging> <name>Flink Quickstart Job</name> <properties> <maven.compiler.source>1.8</maven.compiler.source> <maven.compiler.target>1.8</maven.compiler.target> <flink.version>1.18.1</flink.version> <java.version>1.8</java.version> <scala.binary.version>2.12</scala.binary.version> <kafka.version>3.2.0</kafka.version> <hadoop.version>3.3.4</hadoop.version> <log4j.version>2.16.0</log4j.version> <shadeVersion>3.2.0</shadeVersion> </properties> <dependencies> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-java</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-streaming-java --> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-java</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-clients --> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-clients</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-connector-files --> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-files</artifactId> <version>${flink.version}</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-kafka</artifactId> <version>3.1.0-1.18</version> </dependency> <!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-simple --> <dependency> <groupId>org.apache.logging.log4j</groupId> <artifactId>log4j-slf4j-impl</artifactId> <version>${log4j.version}</version> <scope>runtime</scope> <exclusions> <exclusion> <artifactId>slf4j-api</artifactId> <groupId>org.slf4j</groupId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.logging.log4j</groupId> <artifactId>log4j-api</artifactId> <version>${log4j.version}</version> <scope>runtime</scope> </dependency> <dependency> <groupId>org.apache.logging.log4j</groupId> <artifactId>log4j-core</artifactId> <version>${log4j.version}</version> <scope>runtime</scope> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-azure-fs-hadoop</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-assembly-plugin</artifactId> <version>3.0.0</version> <configuration> <appendAssemblyId>false</appendAssemblyId> <descriptorRefs> <descriptorRef>jar-with-dependencies</descriptorRef> </descriptorRefs> </configuration> <executions> <execution> <id>make-assembly</id> <phase>package</phase> <goals> <goal>single</goal> </goals> </execution> </executions> </plugin> </plugins> </build> </project> And like my reply in stackoverflow, I found the hadoop-common file : https://github.com/apache/hadoop/blob/release-3.3.4-RC1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java#L3374 do not load any filesystem, dig in ServiceLoader.load(FileSystem.class) source code, it looks like have different class loader make it not load any filesystem. I changed the ServiceLoader.load(FileSystem.class) to ServiceLoader.load(FileSystem.class, FileSystem.class.getClassLoader()) and replace the flink-fs-azure-hadoop plugin, it works now, So I'm not sure why it works Gabor Somogyi <gabor.g.somo...@gmail.com> 于2024年6月26日周三 16:52写道： > Hi Xiao, > > I'm not quite convinced that the azure plugin ruined your workload, I > would take a look at the dependency graph you've in the pom. > Adding multiple deps can conflict in terms of class loader services > (src/main/resources/META-INF/services). > > As an example you've 2 such dependencies where > org.apache.flink.core.fs.FileSystemFactory is in the jar. > Hadoop core contains "flie" and the other one something different. Let's > say you don't use service merge plugin in your > maven project. Such case Hadoop core `file` entry will be just overwritten > by the second one. > > Solution: Either avoid deps with conflicting services or add > ServicesResourceTransformer > to your maven project. > > G > > > On Wed, Jun 26, 2024 at 10:16 AM Xiao Xu <ffxrqy...@gmail.com> wrote: > >> Hi, all >> >> I try to use Flink to write Azure Blob Storage which called ADLS, I put >> the flink-azure-fs-hadoop jar in plugins directory and when I start my >> write job it shows: >> >> Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No >> FileSystem for scheme "file" >> at >> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3443) >> ~[hadoop-common-3.3.4.5.1.5.3.jar:?] >> at >> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466) >> ~[hadoop-common-3.3.4.5.1.5.3.jar:?] >> at >> org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174) >> ~[hadoop-common-3.3.4.5.1.5.3.jar:?] >> at >> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574) >> ~[hadoop-common-3.3.4.5.1.5.3.jar:?] >> at >> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521) >> ~[hadoop-common-3.3.4.5.1.5.3.jar:?] >> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540) >> ~[hadoop-common-3.3.4.5.1.5.3.jar:?] >> at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:496) >> ~[hadoop-common-3.3.4.5.1.5.3.jar:?] >> at >> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:316) >> ~[hadoop-common-3.3.4.5.1.5.3.jar:?] >> at >> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:393) >> ~[hadoop-common-3.3.4.5.1.5.3.jar:?] >> at >> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:165) >> ~[hadoop-common-3.3.4.5.1.5.3.jar:?] >> at >> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:146) >> ~[hadoop-common-3.3.4.5.1.5.3.jar:?] >> at >> org.apache.hadoop.fs.store.DataBlocks$DiskBlockFactory.createTmpFileForWrite(DataBlocks.java:980) >> ~[hadoop-common-3.3.4.5.1.5.3.jar:?] >> at >> org.apache.hadoop.fs.store.DataBlocks$DiskBlockFactory.create(DataBlocks.java:960) >> ~[hadoop-common-3.3.4.5.1.5.3.jar:?] >> at >> org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream.createBlockIfNeeded(AbfsOutputStream.java:262) >> ~[hadoop-azure-3.3.4.5.1.5.3.jar:?] >> at >> org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream.<init>(AbfsOutputStream.java:173) >> ~[hadoop-azure-3.3.4.5.1.5.3.jar:?] >> at >> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.createFile(AzureBlobFileSystemStore.java:580) >> ~[hadoop-azure-3.3.4.5.1.5.3.jar:?] >> at >> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.create(AzureBlobFileSystem.java:301) >> ~[hadoop-azure-3.3.4.5.1.5.3.jar:?] >> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1195) >> ~[hadoop-common-3.3.4.5.1.5.3.jar:?] >> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1175) >> ~[hadoop-common-3.3.4.5.1.5.3.jar:?] >> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1064) >> ~[hadoop-common-3.3.4.5.1.5.3.jar:?] >> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1052) >> ~[hadoop-common-3.3.4.5.1.5.3.jar:?] >> >> I search the issue looks like this: >> https://stackoverflow.com/questions/77238642/apache-flink-azure-abfs-file-sink-error-streaming-unsupportedfilesystemexcep >> >> my env: >> Flink: 1.18.1 >> JDK: 1.8 >> >> Does anyone else have the same problem？ >> >
