Hi, I solved my second issue - I was not following Maven's convention for placing source code (I had not placed my source in src/main/java). However, I still would like some help with my first question - what is the recommended way to set a project with multiple main() classes? At the end, I would like to be able to run each main() class as a separate job. Should I create a single JAR and specify different entrypoint classes each time or should I create separate JARs for each main() class?
On Mon, Aug 31, 2020 at 11:13 AM Manas Kale <manaskal...@gmail.com> wrote: > Hi, > I have an IntelliJ project that has multiple classes with main() > functions. I want to package this project as a JAR that I can submit to the > Flink cluster and specify the entry class when I start the job. Here are my > questions: > > - I am not really familiar with Maven and would appreciate some > pointers/examples. From what I understand, I will need to use some sort of > transformer in the Maven shade plugin to merge all of the classes. *If > this is correct, can I see a small example? * > - Also, I can't get a single main class working: > > > <project xmlns="http://maven.apache.org/POM/4.0.0"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 > http://maven.apache.org/xsd/maven-4.0.0.xsd";> > <modelVersion>4.0.0</modelVersion> > > <groupId>flink_summarization</groupId> > <artifactId>flink_summarization</artifactId> > <version>0.1</version> > <packaging>jar</packaging> > > <name>Flink Quickstart Job</name> > <url>http://www.myorganization.org</url> > > <properties> > <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> > <flink.version>1.10.1</flink.version> > <java.version>1.8</java.version> > <scala.binary.version>2.11</scala.binary.version> > <maven.compiler.source>${java.version}</maven.compiler.source> > <maven.compiler.target>${java.version}</maven.compiler.target> > </properties> > > <repositories> > <repository> > <id>apache.snapshots</id> > <name>Apache Development Snapshot Repository</name> > > <url>https://repository.apache.org/content/repositories/snapshots/</url> > <releases> > <enabled>false</enabled> > </releases> > <snapshots> > <enabled>true</enabled> > </snapshots> > </repository> > </repositories> > > <dependencies> > <!-- Apache Flink dependencies --> > <!-- These dependencies are provided, because they should not be > packaged into the JAR file. --> > <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-jdbc --> > <dependency> > <groupId>org.apache.flink</groupId> > <artifactId>flink-java</artifactId> > <version>${flink.version}</version> > <scope>provided</scope> > </dependency> > <dependency> > <groupId>org.apache.flink</groupId> > <artifactId>flink-streaming-java_${scala.binary.version}</artifactId> > <version>${flink.version}</version> > <scope>provided</scope> > </dependency> > > <dependency> > <groupId>org.apache.flink</groupId> > <artifactId>flink-connector-kafka_2.11</artifactId> > <version>${flink.version}</version> > </dependency> > > <dependency> > <groupId>org.apache.flink</groupId> > <artifactId>flink-state-processor-api_2.11</artifactId> > <version>${flink.version}</version> > <scope>provided</scope> > </dependency> > > <dependency> > <groupId>org.apache.flink</groupId> > <artifactId>flink-connector-jdbc_2.11</artifactId> > <version>1.11.0</version> > </dependency> > > <!-- Add logging framework, to produce console output when running in > the IDE. --> > <!-- These dependencies are excluded from the application JAR by > default. --> > <dependency> > <groupId>org.slf4j</groupId> > <artifactId>slf4j-log4j12</artifactId> > <version>1.7.7</version> > <scope>runtime</scope> > </dependency> > <dependency> > <groupId>log4j</groupId> > <artifactId>log4j</artifactId> > <version>1.2.17</version> > <scope>runtime</scope> > </dependency> > > <!-- Test dependencies--> > <dependency> > <groupId>org.apache.flink</groupId> > <artifactId>flink-test-utils_${scala.binary.version}</artifactId> > <version>${flink.version}</version> > <scope>test</scope> > </dependency> > <dependency> > <groupId>org.apache.flink</groupId> > <artifactId>flink-runtime_2.11</artifactId> > <version>${flink.version}</version> > <scope>test</scope> > <classifier>tests</classifier> > </dependency> > <dependency> > <groupId>org.apache.flink</groupId> > <artifactId>flink-streaming-java_2.11</artifactId> > <version>${flink.version}</version> > <scope>test</scope> > <classifier>tests</classifier> > </dependency> > <dependency> > <groupId>org.assertj</groupId> > <artifactId>assertj-core</artifactId> > <!-- use 2.9.1 for Java 7 projects --> > <version>3.16.1</version> > <scope>test</scope> > </dependency> > > > </dependencies> > > <build> > <plugins> > > <!-- Java Compiler --> > <plugin> > <groupId>org.apache.maven.plugins</groupId> > <artifactId>maven-compiler-plugin</artifactId> > <version>3.1</version> > <configuration> > <source>${java.version}</source> > <target>${java.version}</target> > </configuration> > </plugin> > > <!-- We use the maven-shade plugin to create a fat jar that contains > all necessary dependencies. --> > <!-- Change the value of <mainClass>...</mainClass> if your program > entry point changes. --> > <plugin> > <groupId>org.apache.maven.plugins</groupId> > <artifactId>maven-shade-plugin</artifactId> > <version>3.0.0</version> > <!-- author = Manas - dependencyReducedPOM was disabled to stop > maven from entering an infinite loop --> > <configuration> > <createDependencyReducedPom>false</createDependencyReducedPom> > </configuration> > <executions> > <!-- Run shade goal on package phase --> > <execution> > <phase>package</phase> > <goals> > <goal>shade</goal> > </goals> > <configuration> > <artifactSet> > <excludes> > <exclude>org.apache.flink:force-shading</exclude> > <exclude>com.google.code.findbugs:jsr305</exclude> > <exclude>org.slf4j:*</exclude> > <exclude>log4j:*</exclude> > </excludes> > </artifactSet> > <filters> > <filter> > <!-- Do not copy the signatures in the META-INF > folder. > Otherwise, this might cause SecurityExceptions > when using the JAR. --> > <artifact>*:*</artifact> > <excludes> > <exclude>META-INF/*.SF</exclude> > <exclude>META-INF/*.DSA</exclude> > <exclude>META-INF/*.RSA</exclude> > </excludes> > </filter> > </filters> > <transformers> > <transformer > implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> > > <mainClass>iu.feature_summarization.basic_features.pre.BasicPreProcessJob</mainClass> > </transformer> > > </transformers> > </configuration> > </execution> > </executions> > </plugin> > </plugins> > > <pluginManagement> > <plugins> > > <!-- This improves the out-of-the-box experience in Eclipse by > resolving some warnings. --> > <plugin> > <groupId>org.eclipse.m2e</groupId> > <artifactId>lifecycle-mapping</artifactId> > <version>1.0.0</version> > <configuration> > <lifecycleMappingMetadata> > <pluginExecutions> > <pluginExecution> > <pluginExecutionFilter> > <groupId>org.apache.maven.plugins</groupId> > <artifactId>maven-shade-plugin</artifactId> > <versionRange>[3.0.0,)</versionRange> > <goals> > <goal>shade</goal> > </goals> > </pluginExecutionFilter> > <action> > <ignore/> > </action> > </pluginExecution> > <pluginExecution> > <pluginExecutionFilter> > <groupId>org.apache.maven.plugins</groupId> > <artifactId>maven-compiler-plugin</artifactId> > <versionRange>[3.1,)</versionRange> > <goals> > <goal>testCompile</goal> > <goal>compile</goal> > </goals> > </pluginExecutionFilter> > <action> > <ignore/> > </action> > </pluginExecution> > </pluginExecutions> > </lifecycleMappingMetadata> > </configuration> > </plugin> > </plugins> > </pluginManagement> > </build> > > <!-- This profile helps to make things run out of the box in IntelliJ --> > <!-- Its adds Flink's core classes to the runtime class path. --> > <!-- Otherwise they are missing in IntelliJ, because the dependency is > 'provided' --> > <profiles> > <profile> > <id>add-dependencies-for-IDEA</id> > > <activation> > <property> > <name>idea.version</name> > </property> > </activation> > > <dependencies> > <dependency> > <groupId>org.apache.flink</groupId> > <artifactId>flink-java</artifactId> > <version>${flink.version}</version> > <scope>compile</scope> > </dependency> > <dependency> > <groupId>org.apache.flink</groupId> > > <artifactId>flink-streaming-java_${scala.binary.version}</artifactId> > <version>${flink.version}</version> > <scope>compile</scope> > </dependency> > </dependencies> > </profile> > </profiles> > > </project> > > > The fully qualified name within tag <mainClass> is correct and contains > the main() method. However, when I submit this to Flink I get the error: > > ../../flink-1.11.0/bin/flink run target/flink_summarization-0.1.jar > > ------------------------------------------------------------ > The program finished with the following exception: > > org.apache.flink.client.program.ProgramInvocationException: The program's > entry point class ' > iu.feature_summarization.basic_features.pre.BasicPreProcessJob' was not > found in the jar file > . > > *Am I specifying some <scope> tag incorrectly in my pom file?* > > > Thank you, > Manas >
