Hi, I'm a student and I using flink v1.2.1 to develop a workflow to analyze data coming from kafka v0.10.
At the moment everything work fine, from eclipse I can export the runnable jar with dependencies and with "sudo java -jar flinkApp.jar" I can run the flink application on ubuntu server. Until here all ok. I configured flink to be in High Avaliabity mode, it's start correcly. When I try to export with maven, it create a fat jar, i launch that with "sudo bin/flink run flinkApp.jar". It's work and on the dashboard I can the see the job is running. The problem is that is not receive a single data from kafka. The cuorious thing is that only when I export the jar in order to run in cluster mode I've this problem, whereas I export a runnable jar everything work fine. I put in attachment three files: - webUI_producer: it shows the flink dashboard with my kafka producer - problemFlinkCluster: it shows the ubuntu server where my flink application run. The top left square is zookeeper (from kafka), the bottom left is the flink program running with "sudo java -jar flinkapp.jar" and the right most column is the execution of flink app with "sudo bin/flink run appFlink.jar". You can see The flink application start and than do nothing more. - the pom.xml I think the problem is in the pom.xml but I don't know where. Best regards, simone
<project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> <modelVersion>4.0.0</modelVersion> <groupId>ch.supsi.dti</groupId> <artifactId>LogAnalyzerCluster</artifactId> <version>0.0.1-SNAPSHOT</version> <packaging>jar</packaging> <name>LogAnalyzerCluster</name> <url>http://maven.apache.org</url> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <maven.compiler.source>1.8</maven.compiler.source> <maven.compiler.target>1.8</maven.compiler.target> <flink.version>1.2.1</flink.version> <mainClass.name>ch.supsi.dti.WordCountFlink.App</mainClass.name> </properties> <dependencies> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>3.8.1</version> <scope>test</scope> </dependency> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-nop</artifactId> <version>1.7.25</version> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> </exclusions> </dependency> <!-- https://mvnrepository.com/artifact/com.esotericsoftware.kryo/kryo --> <dependency> <groupId>com.esotericsoftware.kryo</groupId> <artifactId>kryo</artifactId> <version>2.24.0</version> </dependency> <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-core --> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-core</artifactId> <version>${flink.version}</version> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> </exclusions> </dependency> <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-streaming-java_2.11 --> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-java_2.11</artifactId> <version>${flink.version}</version> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> </exclusions> </dependency> <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka-0.10_2.11 --> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-kafka-0.10_2.11</artifactId> <version>${flink.version}</version> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> </exclusions> </dependency> <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-cep_2.11 --> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-cep_2.11</artifactId> <version>${flink.version}</version> </dependency> <!-- https://mvnrepository.com/artifact/net.sf.opencsv/opencsv --> <dependency> <groupId>net.sf.opencsv</groupId> <artifactId>opencsv</artifactId> <version>2.3</version> </dependency> <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-connector-cassandra_2.11 --> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-cassandra_2.11</artifactId> <version>${flink.version}</version> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> </exclusions> </dependency> <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-table_2.11 --> <!-- <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-table_2.11</artifactId> <version>${flink.version}</version> </dependency> --> <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-clients_2.11 --> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-clients_2.11</artifactId> <version>${flink.version}</version> </dependency> </dependencies> <profiles> <profile> <!-- Profile for packaging correct JAR files --> <id>build-jar</id> <activation> <activeByDefault>false</activeByDefault> </activation> <dependencies> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-java</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-java_2.11</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-clients_2.11</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <!-- <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> <version>${slf4j.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>log4j</groupId> <artifactId>log4j</artifactId> <version>${log4j.version}</version> <scope>provided</scope> </dependency> --> </dependencies> <build> <plugins> <!-- disable the exclusion rules --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>2.4.1</version> <executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <artifactSet> <excludes combine.self="override"></excludes> </artifactSet> <!-- <relocations> <relocation> <pattern>com.google.common</pattern> <shadedPattern>com.shaded.google.common</shadedPattern> </relocation> </relocations> --> </configuration> </execution> </executions> </plugin> </plugins> </build> </profile> </profiles> <build> <plugins> <!-- We use the maven-shade plugin to create a fat jar that contains all dependencies except flink and its transitive dependencies. The resulting fat-jar can be executed on a cluster. Change the value of Program-Class if your program entry point changes. --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>2.4.1</version> <executions> <!-- Run shade goal on package phase --> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <artifactSet> <excludes> <!-- This list contains all dependencies of flink-dist Everything else will be packaged into the fat-jar --> <exclude>org.apache.flink:flink-annotations</exclude> <exclude>org.apache.flink:flink-shaded-hadoop2</exclude> <exclude>org.apache.flink:flink-shaded-curator-recipes</exclude> <exclude>org.apache.flink:flink-core</exclude> <exclude>org.apache.flink:flink-java</exclude> <exclude>org.apache.flink:flink-scala_2.11</exclude> <exclude>org.apache.flink:flink-runtime_2.11</exclude> <exclude>org.apache.flink:flink-optimizer_2.11</exclude> <exclude>org.apache.flink:flink-clients_2.11</exclude> <exclude>org.apache.flink:flink-avro_2.11</exclude> <exclude>org.apache.flink:flink-examples-batch_2.11</exclude> <exclude>org.apache.flink:flink-examples-streaming_2.11</exclude> <exclude>org.apache.flink:flink-streaming-java_2.11</exclude> <exclude>org.apache.flink:flink-streaming-scala_2.11</exclude> <exclude>org.apache.flink:flink-scala-shell_2.11</exclude> <exclude>org.apache.flink:flink-python</exclude> <exclude>org.apache.flink:flink-metrics-core</exclude> <exclude>org.apache.flink:flink-metrics-jmx</exclude> <exclude>org.apache.flink:flink-statebackend-rocksdb_2.11</exclude> <!-- Also exclude very big transitive dependencies of Flink WARNING: You have to remove these excludes if your code relies on other versions of these dependencies. --> <exclude>log4j:log4j</exclude> <exclude>org.scala-lang:scala-library</exclude> <exclude>org.scala-lang:scala-compiler</exclude> <exclude>org.scala-lang:scala-reflect</exclude> <exclude>com.data-artisans:flakka-actor_*</exclude> <exclude>com.data-artisans:flakka-remote_*</exclude> <exclude>com.data-artisans:flakka-slf4j_*</exclude> <exclude>io.netty:netty-all</exclude> <exclude>io.netty:netty</exclude> <exclude>commons-fileupload:commons-fileupload</exclude> <exclude>org.apache.avro:avro</exclude> <exclude>commons-collections:commons-collections</exclude> <exclude>org.codehaus.jackson:jackson-core-asl</exclude> <exclude>org.codehaus.jackson:jackson-mapper-asl</exclude> <exclude>com.thoughtworks.paranamer:paranamer</exclude> <exclude>org.xerial.snappy:snappy-java</exclude> <exclude>org.apache.commons:commons-compress</exclude> <exclude>org.tukaani:xz</exclude> <exclude>com.esotericsoftware.kryo:kryo</exclude> <exclude>com.esotericsoftware.minlog:minlog</exclude> <exclude>org.objenesis:objenesis</exclude> <exclude>com.twitter:chill_*</exclude> <exclude>com.twitter:chill-java</exclude> <exclude>commons-lang:commons-lang</exclude> <exclude>junit:junit</exclude> <exclude>org.apache.commons:commons-lang3</exclude> <exclude>org.slf4j:slf4j-api</exclude> <exclude>org.slf4j:slf4j-log4j12</exclude> <exclude>log4j:log4j</exclude> <exclude>org.apache.commons:commons-math</exclude> <exclude>org.apache.sling:org.apache.sling.commons.json</exclude> <exclude>commons-logging:commons-logging</exclude> <exclude>commons-codec:commons-codec</exclude> <exclude>com.fasterxml.jackson.core:jackson-core</exclude> <exclude>com.fasterxml.jackson.core:jackson-databind</exclude> <exclude>com.fasterxml.jackson.core:jackson-annotations</exclude> <exclude>stax:stax-api</exclude> <exclude>com.typesafe:config</exclude> <exclude>org.uncommons.maths:uncommons-maths</exclude> <exclude>com.github.scopt:scopt_*</exclude> <exclude>commons-io:commons-io</exclude> <exclude>commons-cli:commons-cli</exclude> </excludes> </artifactSet> <filters> <filter> <artifact>org.apache.flink:*</artifact> <excludes> <!-- exclude shaded google but include shaded curator --> <exclude>org/apache/flink/shaded/com/**</exclude> <exclude>web-docs/**</exclude> </excludes> </filter> <filter> <!-- Do not copy the signatures in the META-INF folder. Otherwise, this might cause SecurityExceptions when using the JAR. --> <artifact>*:*</artifact> <excludes> <exclude>META-INF/*.SF</exclude> <exclude>META-INF/*.DSA</exclude> <exclude>META-INF/*.RSA</exclude> </excludes> </filter> </filters> <!-- If you want to use ./bin/flink run <quickstart jar> uncomment the following lines. This will add a Main-Class entry to the manifest file --> <!-- --> <transformers> <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> <mainClass>${mainClass.name}</mainClass> </transformer> </transformers> <createDependencyReducedPom>false</createDependencyReducedPom> </configuration> </execution> </executions> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>3.1</version> <configuration> <source>1.8</source> <!-- If you want to use Java 8, change this to "1.8" --> <target>1.8</target> <!-- If you want to use Java 8, change this to "1.8" --> </configuration> </plugin> </plugins> <!-- If you want to use Java 8 Lambda Expressions uncomment the following lines --> <!-- --> <pluginManagement> <plugins> <plugin> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>1.8</source> <target>1.8</target> <compilerId>jdt</compilerId> </configuration> <dependencies> <dependency> <groupId>org.eclipse.tycho</groupId> <artifactId>tycho-compiler-jdt</artifactId> <version>0.21.0</version> </dependency> </dependencies> </plugin> <plugin> <groupId>org.eclipse.m2e</groupId> <artifactId>lifecycle-mapping</artifactId> <version>1.0.0</version> <configuration> <lifecycleMappingMetadata> <pluginExecutions> <pluginExecution> <pluginExecutionFilter> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-assembly-plugin</artifactId> <versionRange>[2.4,)</versionRange> <goals> <goal>single</goal> </goals> </pluginExecutionFilter> <action> <ignore/> </action> </pluginExecution> <pluginExecution> <pluginExecutionFilter> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <versionRange>[3.1,)</versionRange> <goals> <goal>testCompile</goal> <goal>compile</goal> </goals> </pluginExecutionFilter> <action> <ignore/> </action> </pluginExecution> </pluginExecutions> </lifecycleMappingMetadata> </configuration> </plugin> </plugins> </pluginManagement> </build> </project>
