Guess I figured out a solution for the first question as well - I am packaging multiple main() classes in the same JAR and specifying entrypoint classes when submitting the JAR. Most of my issues stemmed from an improperly configured POM file and a mismatch in Flink runtime versions. I'll assume this is the recommended way to go about doing this, thanks for reading and have a great day!
On Mon, Aug 31, 2020 at 12:03 PM Manas Kale <manaskal...@gmail.com> wrote: > Hi, > I solved my second issue - I was not following Maven's convention for > placing source code (I had not placed my source in src/main/java). > However, I still would like some help with my first question - what is the > recommended way to set a project with multiple main() classes? At the end, > I would like to be able to run each main() class as a separate job. Should > I create a single JAR and specify different entrypoint classes each time or > should I create separate JARs for each main() class? > > On Mon, Aug 31, 2020 at 11:13 AM Manas Kale <manaskal...@gmail.com> wrote: > >> Hi, >> I have an IntelliJ project that has multiple classes with main() >> functions. I want to package this project as a JAR that I can submit to the >> Flink cluster and specify the entry class when I start the job. Here are my >> questions: >> >> - I am not really familiar with Maven and would appreciate some >> pointers/examples. From what I understand, I will need to use some sort of >> transformer in the Maven shade plugin to merge all of the classes. *If >> this is correct, can I see a small example? * >> - Also, I can't get a single main class working: >> >> >> <project xmlns="http://maven.apache.org/POM/4.0.0"; >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; >> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 >> http://maven.apache.org/xsd/maven-4.0.0.xsd";> >> <modelVersion>4.0.0</modelVersion> >> >> <groupId>flink_summarization</groupId> >> <artifactId>flink_summarization</artifactId> >> <version>0.1</version> >> <packaging>jar</packaging> >> >> <name>Flink Quickstart Job</name> >> <url>http://www.myorganization.org</url> >> >> <properties> >> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> >> <flink.version>1.10.1</flink.version> >> <java.version>1.8</java.version> >> <scala.binary.version>2.11</scala.binary.version> >> <maven.compiler.source>${java.version}</maven.compiler.source> >> <maven.compiler.target>${java.version}</maven.compiler.target> >> </properties> >> >> <repositories> >> <repository> >> <id>apache.snapshots</id> >> <name>Apache Development Snapshot Repository</name> >> >> <url>https://repository.apache.org/content/repositories/snapshots/</url> >> <releases> >> <enabled>false</enabled> >> </releases> >> <snapshots> >> <enabled>true</enabled> >> </snapshots> >> </repository> >> </repositories> >> >> <dependencies> >> <!-- Apache Flink dependencies --> >> <!-- These dependencies are provided, because they should not be >> packaged into the JAR file. --> >> <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-jdbc --> >> <dependency> >> <groupId>org.apache.flink</groupId> >> <artifactId>flink-java</artifactId> >> <version>${flink.version}</version> >> <scope>provided</scope> >> </dependency> >> <dependency> >> <groupId>org.apache.flink</groupId> >> >> <artifactId>flink-streaming-java_${scala.binary.version}</artifactId> >> <version>${flink.version}</version> >> <scope>provided</scope> >> </dependency> >> >> <dependency> >> <groupId>org.apache.flink</groupId> >> <artifactId>flink-connector-kafka_2.11</artifactId> >> <version>${flink.version}</version> >> </dependency> >> >> <dependency> >> <groupId>org.apache.flink</groupId> >> <artifactId>flink-state-processor-api_2.11</artifactId> >> <version>${flink.version}</version> >> <scope>provided</scope> >> </dependency> >> >> <dependency> >> <groupId>org.apache.flink</groupId> >> <artifactId>flink-connector-jdbc_2.11</artifactId> >> <version>1.11.0</version> >> </dependency> >> >> <!-- Add logging framework, to produce console output when running in >> the IDE. --> >> <!-- These dependencies are excluded from the application JAR by >> default. --> >> <dependency> >> <groupId>org.slf4j</groupId> >> <artifactId>slf4j-log4j12</artifactId> >> <version>1.7.7</version> >> <scope>runtime</scope> >> </dependency> >> <dependency> >> <groupId>log4j</groupId> >> <artifactId>log4j</artifactId> >> <version>1.2.17</version> >> <scope>runtime</scope> >> </dependency> >> >> <!-- Test dependencies--> >> <dependency> >> <groupId>org.apache.flink</groupId> >> <artifactId>flink-test-utils_${scala.binary.version}</artifactId> >> <version>${flink.version}</version> >> <scope>test</scope> >> </dependency> >> <dependency> >> <groupId>org.apache.flink</groupId> >> <artifactId>flink-runtime_2.11</artifactId> >> <version>${flink.version}</version> >> <scope>test</scope> >> <classifier>tests</classifier> >> </dependency> >> <dependency> >> <groupId>org.apache.flink</groupId> >> <artifactId>flink-streaming-java_2.11</artifactId> >> <version>${flink.version}</version> >> <scope>test</scope> >> <classifier>tests</classifier> >> </dependency> >> <dependency> >> <groupId>org.assertj</groupId> >> <artifactId>assertj-core</artifactId> >> <!-- use 2.9.1 for Java 7 projects --> >> <version>3.16.1</version> >> <scope>test</scope> >> </dependency> >> >> >> </dependencies> >> >> <build> >> <plugins> >> >> <!-- Java Compiler --> >> <plugin> >> <groupId>org.apache.maven.plugins</groupId> >> <artifactId>maven-compiler-plugin</artifactId> >> <version>3.1</version> >> <configuration> >> <source>${java.version}</source> >> <target>${java.version}</target> >> </configuration> >> </plugin> >> >> <!-- We use the maven-shade plugin to create a fat jar that >> contains all necessary dependencies. --> >> <!-- Change the value of <mainClass>...</mainClass> if your program >> entry point changes. --> >> <plugin> >> <groupId>org.apache.maven.plugins</groupId> >> <artifactId>maven-shade-plugin</artifactId> >> <version>3.0.0</version> >> <!-- author = Manas - dependencyReducedPOM was disabled to stop >> maven from entering an infinite loop --> >> <configuration> >> <createDependencyReducedPom>false</createDependencyReducedPom> >> </configuration> >> <executions> >> <!-- Run shade goal on package phase --> >> <execution> >> <phase>package</phase> >> <goals> >> <goal>shade</goal> >> </goals> >> <configuration> >> <artifactSet> >> <excludes> >> <exclude>org.apache.flink:force-shading</exclude> >> <exclude>com.google.code.findbugs:jsr305</exclude> >> <exclude>org.slf4j:*</exclude> >> <exclude>log4j:*</exclude> >> </excludes> >> </artifactSet> >> <filters> >> <filter> >> <!-- Do not copy the signatures in the META-INF >> folder. >> Otherwise, this might cause SecurityExceptions >> when using the JAR. --> >> <artifact>*:*</artifact> >> <excludes> >> <exclude>META-INF/*.SF</exclude> >> <exclude>META-INF/*.DSA</exclude> >> <exclude>META-INF/*.RSA</exclude> >> </excludes> >> </filter> >> </filters> >> <transformers> >> <transformer >> implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> >> >> <mainClass>iu.feature_summarization.basic_features.pre.BasicPreProcessJob</mainClass> >> </transformer> >> >> </transformers> >> </configuration> >> </execution> >> </executions> >> </plugin> >> </plugins> >> >> <pluginManagement> >> <plugins> >> >> <!-- This improves the out-of-the-box experience in Eclipse by >> resolving some warnings. --> >> <plugin> >> <groupId>org.eclipse.m2e</groupId> >> <artifactId>lifecycle-mapping</artifactId> >> <version>1.0.0</version> >> <configuration> >> <lifecycleMappingMetadata> >> <pluginExecutions> >> <pluginExecution> >> <pluginExecutionFilter> >> <groupId>org.apache.maven.plugins</groupId> >> <artifactId>maven-shade-plugin</artifactId> >> <versionRange>[3.0.0,)</versionRange> >> <goals> >> <goal>shade</goal> >> </goals> >> </pluginExecutionFilter> >> <action> >> <ignore/> >> </action> >> </pluginExecution> >> <pluginExecution> >> <pluginExecutionFilter> >> <groupId>org.apache.maven.plugins</groupId> >> <artifactId>maven-compiler-plugin</artifactId> >> <versionRange>[3.1,)</versionRange> >> <goals> >> <goal>testCompile</goal> >> <goal>compile</goal> >> </goals> >> </pluginExecutionFilter> >> <action> >> <ignore/> >> </action> >> </pluginExecution> >> </pluginExecutions> >> </lifecycleMappingMetadata> >> </configuration> >> </plugin> >> </plugins> >> </pluginManagement> >> </build> >> >> <!-- This profile helps to make things run out of the box in IntelliJ --> >> <!-- Its adds Flink's core classes to the runtime class path. --> >> <!-- Otherwise they are missing in IntelliJ, because the dependency is >> 'provided' --> >> <profiles> >> <profile> >> <id>add-dependencies-for-IDEA</id> >> >> <activation> >> <property> >> <name>idea.version</name> >> </property> >> </activation> >> >> <dependencies> >> <dependency> >> <groupId>org.apache.flink</groupId> >> <artifactId>flink-java</artifactId> >> <version>${flink.version}</version> >> <scope>compile</scope> >> </dependency> >> <dependency> >> <groupId>org.apache.flink</groupId> >> >> <artifactId>flink-streaming-java_${scala.binary.version}</artifactId> >> <version>${flink.version}</version> >> <scope>compile</scope> >> </dependency> >> </dependencies> >> </profile> >> </profiles> >> >> </project> >> >> >> The fully qualified name within tag <mainClass> is correct and contains >> the main() method. However, when I submit this to Flink I get the error: >> > ../../flink-1.11.0/bin/flink run target/flink_summarization-0.1.jar >> >> ------------------------------------------------------------ >> The program finished with the following exception: >> >> org.apache.flink.client.program.ProgramInvocationException: The program's >> entry point class ' >> iu.feature_summarization.basic_features.pre.BasicPreProcessJob' was not >> found in the jar file >> . >> >> *Am I specifying some <scope> tag incorrectly in my pom file?* >> >> >> Thank you, >> Manas >> >
