Yep the same issue as before(class not found) with flink 0.10.2 with scala version 2.11. I was not able to use scala 2.10 since connector for flink_connector_kafka for 0.10.2 is not available.
balaji On Mon, Mar 14, 2016 at 4:20 PM, Balaji Rajagopalan < balaji.rajagopa...@olacabs.com> wrote: > Yes figured that out, thanks for point that, my bad. I have put back > 0.10.2 as flink version, will try to reproduce the problem again, this time > I have explicitly called out the scala version as 2.11. > > > On Mon, Mar 14, 2016 at 4:14 PM, Robert Metzger <rmetz...@apache.org> > wrote: > >> Hi, >> >> flink-connector-kafka_ doesn't exist for 1.0.0. You have to use either >> flink-connector-kafka-0.8_ or flink-connector-kafka-0.9_ >> >> >> On Mon, Mar 14, 2016 at 11:17 AM, Balaji Rajagopalan < >> balaji.rajagopa...@olacabs.com> wrote: >> >>> What I noticied was that, if I remove the dependency on >>> flink-connector-kafka so it is clearly to do something with that >>> dependency. >>> >>> >>> On Mon, Mar 14, 2016 at 3:46 PM, Balaji Rajagopalan < >>> balaji.rajagopa...@olacabs.com> wrote: >>> >>>> Robert, >>>> I have moved on to latest version of flink of 1.0.0 hoping that >>>> will solve my problem with kafka connector . Here is my pom.xml but now I >>>> cannot get the code compiled. >>>> >>>> [ERROR] Failed to execute goal >>>> net.alchim31.maven:scala-maven-plugin:3.2.1:compile (scala-compile-first) >>>> on project flink-streaming-demo: Execution scala-compile-first of goal >>>> net.alchim31.maven:scala-maven-plugin:3.2.1:compile failed: For artifact >>>> {null:null:null:jar}: The groupId cannot be empty. -> [Help 1] >>>> >>>> I read about the above errors in most cases people where able to >>>> overcome is by deleting the .m2 directory, and that did not fix the issue >>>> for me. >>>> >>>> What I noticied was that, if I remove the dependency on >>>> >>>> Here is my pom.xml >>>> >>>> <?xml version="1.0" encoding="UTF-8"?> >>>> <!-- >>>> Copyright 2015 data Artisans GmbH >>>> >>>> Licensed under the Apache License, Version 2.0 (the "License"); >>>> you may not use this file except in compliance with the License. >>>> You may obtain a copy of the License at >>>> >>>> http://www.apache.org/licenses/LICENSE-2.0 >>>> >>>> Unless required by applicable law or agreed to in writing, software >>>> distributed under the License is distributed on an "AS IS" BASIS, >>>> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. >>>> See the License for the specific language governing permissions and >>>> limitations under the License. >>>> --> >>>> <project xmlns="http://maven.apache.org/POM/4.0.0" >>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >>>> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 >>>> http://maven.apache.org/xsd/maven-4.0.0.xsd"> >>>> <modelVersion>4.0.0</modelVersion> >>>> >>>> <groupId>com.dataArtisans</groupId> >>>> <artifactId>flink-streaming-demo</artifactId> >>>> <version>0.1</version> >>>> <packaging>jar</packaging> >>>> >>>> <name>Flink Streaming Demo</name> >>>> <url>http://www.data-artisans.com</url> >>>> >>>> <properties> >>>> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> >>>> <slf4j.version>1.7.12</slf4j.version> >>>> <flink.version>1.0.0</flink.version> >>>> <scala.version>2.10</scala.version> >>>> </properties> >>>> >>>> <dependencies> >>>> >>>> >>>> >>>> <dependency> >>>> <groupId>org.apache.flink</groupId> >>>> <artifactId>flink-streaming-scala_${scala.version}</artifactId> >>>> <version>${flink.version}</version> >>>> </dependency> >>>> >>>> <dependency> >>>> <groupId>org.apache.flink</groupId> >>>> <artifactId>flink-runtime-web_${scala.version}</artifactId> >>>> <version>${flink.version}</version> >>>> </dependency> >>>> >>>> <dependency> >>>> <groupId>org.elasticsearch</groupId> >>>> <artifactId>elasticsearch</artifactId> >>>> <version>1.7.3</version> >>>> <scope>compile</scope> >>>> </dependency> >>>> >>>> <dependency> >>>> <groupId>joda-time</groupId> >>>> <artifactId>joda-time</artifactId> >>>> <version>2.7</version> >>>> </dependency> >>>> >>>> <dependency> >>>> <groupId>org.apache.kafka</groupId> >>>> <artifactId>kafka_${scala.version}</artifactId> >>>> <version>0.8.2.0</version> >>>> </dependency> >>>> >>>> <dependency> >>>> <groupId>org.apache.flink</groupId> >>>> <artifactId>flink-connector-kafka_${scala.version}</artifactId> >>>> <version>${flink.version}</version> >>>> </dependency> >>>> >>>> <dependency> >>>> <groupId>org.json4s</groupId> >>>> <artifactId>json4s-native_${scala.version}</artifactId> >>>> <version>3.3.0</version> >>>> </dependency> >>>> >>>> >>>> </dependencies> >>>> >>>> <build> >>>> <plugins> >>>> >>>> <!-- Scala Compiler --> >>>> <plugin> >>>> <groupId>net.alchim31.maven</groupId> >>>> <artifactId>scala-maven-plugin</artifactId> >>>> <version>3.2.1</version> >>>> <executions> >>>> <!-- Run scala compiler in the process-resources phase, so >>>> that dependencies on >>>> scala classes can be resolved later in the (Java) >>>> compile phase --> >>>> <execution> >>>> <id>scala-compile-first</id> >>>> <phase>process-resources</phase> >>>> <goals> >>>> <goal>compile</goal> >>>> </goals> >>>> </execution> >>>> >>>> <!-- Run scala compiler in the process-test-resources >>>> phase, so that dependencies on >>>> scala classes can be resolved later in the (Java) >>>> test-compile phase --> >>>> <execution> >>>> <id>scala-test-compile</id> >>>> <phase>process-test-resources</phase> >>>> <goals> >>>> <goal>testCompile</goal> >>>> </goals> >>>> </execution> >>>> </executions> >>>> <configuration> >>>> <jvmArgs> >>>> <jvmArg>-Xms128m</jvmArg> >>>> <jvmArg>-Xmx512m</jvmArg> >>>> </jvmArgs> >>>> </configuration> >>>> </plugin> >>>> >>>> <plugin> >>>> <groupId>org.apache.maven.plugins</groupId> >>>> <artifactId>maven-dependency-plugin</artifactId> >>>> <version>2.9</version> >>>> <executions> >>>> <execution> >>>> <id>unpack</id> >>>> <!-- executed just before the package phase --> >>>> <phase>prepare-package</phase> >>>> <goals> >>>> <goal>unpack</goal> >>>> </goals> >>>> <configuration> >>>> <artifactItems> >>>> <!-- For Flink connector classes --> >>>> <artifactItem> >>>> <groupId>org.apache.flink</groupId> >>>> >>>> <artifactId>flink-connector-kafka_${scala.version}</artifactId> >>>> <version>1.0.0</version> >>>> <type>jar</type> >>>> <overWrite>false</overWrite> >>>> >>>> <outputDirectory>${project.build.directory}/classes</outputDirectory> >>>> <includes>org/apache/flink/**</includes> >>>> </artifactItem> >>>> <!-- For Kafka API classes --> >>>> <artifactItem> >>>> <groupId>org.apache.kafka</groupId> >>>> <artifactId>kafka_${scala.version}</artifactId> >>>> <version>0.8.2.0</version> >>>> <type>jar</type> >>>> <overWrite>false</overWrite> >>>> >>>> <outputDirectory>${project.build.directory}/classes</outputDirectory> >>>> <includes>kafka/**</includes> >>>> </artifactItem> >>>> </artifactItems> >>>> </configuration> >>>> </execution> >>>> </executions> >>>> </plugin> >>>> >>>> <!--plugin> >>>> <groupId>org.apache.maven.plugins</groupId> >>>> <artifactId>maven-jar-plugin</artifactId> >>>> <executions> >>>> >>>> <execution> >>>> <id>MBoxParser</id> >>>> <phase>package</phase> >>>> <goals> >>>> <goal>jar</goal> >>>> </goals> >>>> >>>> <configuration> >>>> <classifier>MBoxParser</classifier> >>>> >>>> <archive> >>>> <manifestEntries> >>>> >>>> <main-class>com.dataartisans.flinkTraining.dataSetPreparation.MBoxParser</main-class> >>>> </manifestEntries> >>>> </archive> >>>> >>>> <includes> >>>> <include>**/MBoxParser.class</include> >>>> <include>**/MBoxParser$*.class</include> >>>> </includes> >>>> </configuration> >>>> </execution> >>>> >>>> </executions> >>>> </plugin--> >>>> >>>> <plugin> >>>> <groupId>org.apache.maven.plugins</groupId> >>>> <artifactId>maven-compiler-plugin</artifactId> >>>> <version>3.1</version> >>>> <configuration> >>>> <source>1.8</source> <!-- If you want to use Java 8, change >>>> this to "1.8" --> >>>> <target>1.8</target> <!-- If you want to use Java 8, change >>>> this to "1.8" --> >>>> </configuration> >>>> </plugin> >>>> >>>> <plugin> >>>> <groupId>org.apache.rat</groupId> >>>> <artifactId>apache-rat-plugin</artifactId> >>>> <version>0.10</version><!--$NO-MVN-MAN-VER$--> >>>> <inherited>false</inherited> >>>> <executions> >>>> <execution> >>>> <phase>verify</phase> >>>> <goals> >>>> <goal>check</goal> >>>> </goals> >>>> </execution> >>>> </executions> >>>> <configuration> >>>> <excludeSubProjects>false</excludeSubProjects> >>>> <numUnapprovedLicenses>0</numUnapprovedLicenses> >>>> <licenses> >>>> <!-- Enforce this license: >>>> Copyright 2015 data Artisans GmbH >>>> >>>> Licensed under the Apache License, Version 2.0 (the >>>> "License"); >>>> you may not use this file except in compliance with >>>> the License. >>>> You may obtain a copy of the License at >>>> >>>> http://www.apache.org/licenses/LICENSE-2.0 >>>> >>>> Unless required by applicable law or agreed to in >>>> writing, software >>>> distributed under the License is distributed on an >>>> "AS IS" BASIS, >>>> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either >>>> express or implied. >>>> See the License for the specific language governing >>>> permissions and >>>> limitations under the License. >>>> --> >>>> <license >>>> implementation="org.apache.rat.analysis.license.SimplePatternBasedLicense"> >>>> <licenseFamilyCategory>AL2 </licenseFamilyCategory> >>>> <licenseFamilyName>Apache License >>>> 2.0</licenseFamilyName> >>>> <notes /> >>>> <patterns> >>>> <pattern>Copyright 2015 data Artisans >>>> GmbH</pattern> >>>> <pattern>Licensed under the Apache License, >>>> Version 2.0 (the "License");</pattern> >>>> </patterns> >>>> </license> >>>> </licenses> >>>> <licenseFamilies> >>>> <licenseFamily >>>> implementation="org.apache.rat.license.SimpleLicenseFamily"> >>>> <familyName>Apache License 2.0</familyName> >>>> </licenseFamily> >>>> </licenseFamilies> >>>> <excludes> >>>> <!-- Additional files like .gitignore etc.--> >>>> <exclude>**/.*</exclude> >>>> <exclude>**/*.prefs</exclude> >>>> <exclude>**/*.properties</exclude> >>>> <exclude>**/*.log</exclude> >>>> <exclude>*.txt/**</exclude> >>>> <!-- Administrative files in the main trunk. --> >>>> <exclude>**/README.md</exclude> >>>> <exclude>CHANGELOG</exclude> >>>> <!-- Build files --> >>>> <exclude>**/*.iml</exclude> >>>> <!-- Generated content --> >>>> <exclude>**/target/**</exclude> >>>> <exclude>**/build/**</exclude> >>>> </excludes> >>>> </configuration> >>>> </plugin> >>>> >>>> <plugin> >>>> <groupId>org.apache.maven.plugins</groupId> >>>> <artifactId>maven-checkstyle-plugin</artifactId> >>>> <version>2.12.1</version> >>>> <executions> >>>> <execution> >>>> <id>validate</id> >>>> <phase>validate</phase> >>>> <goals> >>>> <goal>check</goal> >>>> </goals> >>>> </execution> >>>> </executions> >>>> <configuration> >>>> <configLocation>/tools/maven/checkstyle.xml</configLocation> >>>> <logViolationsToConsole>true</logViolationsToConsole> >>>> </configuration> >>>> </plugin> >>>> >>>> </plugins> >>>> >>>> </build> >>>> </project> >>>> >>>> >>>> On Mon, Mar 14, 2016 at 3:15 PM, Robert Metzger <rmetz...@apache.org> >>>> wrote: >>>> >>>>> Can you send me the full build file to further investigate the issue? >>>>> >>>>> On Fri, Mar 11, 2016 at 4:56 PM, Balaji Rajagopalan < >>>>> balaji.rajagopa...@olacabs.com> wrote: >>>>> >>>>>> Robert, >>>>>> That did not fix it ( using flink and connector same version) . >>>>>> Tried with scala version 2.11, so will try to see scala 2.10 makes any >>>>>> difference. >>>>>> >>>>>> balaji >>>>>> >>>>>> On Fri, Mar 11, 2016 at 8:06 PM, Robert Metzger <rmetz...@apache.org> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> you have to use the same version for all dependencies from the >>>>>>> "org.apache.flink" group. >>>>>>> >>>>>>> You said these are the versions you are using: >>>>>>> >>>>>>> flink.version = 0.10.2 >>>>>>> kafka.verison = 0.8.2 >>>>>>> flink.kafka.connection.verion=0.9.1 >>>>>>> >>>>>>> For the connector, you also need to use 0.10.2. >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Mar 11, 2016 at 9:56 AM, Balaji Rajagopalan < >>>>>>> balaji.rajagopa...@olacabs.com> wrote: >>>>>>> >>>>>>>> I am tyring to use the flink kafka connector, for this I have >>>>>>>> specified the kafka connector dependency and created a fat jar since >>>>>>>> default flink installation does not contain kafka connector jars. I >>>>>>>> have >>>>>>>> made sure that flink-streaming-demo-0.1.jar has the >>>>>>>> kafka.javaapi.consumer.SimpleConsumer.class but still I see the class >>>>>>>> not >>>>>>>> found exception. >>>>>>>> >>>>>>>> The code for kafka connector in flink. >>>>>>>> >>>>>>>> val env = StreamExecutionEnvironment.getExecutionEnvironment >>>>>>>> val prop:Properties = new Properties() >>>>>>>> prop.setProperty("zookeeper.connect","somezookeer:2181") >>>>>>>> prop.setProperty("group.id","some-group") >>>>>>>> prop.setProperty("bootstrap.servers","somebroker:9092") >>>>>>>> >>>>>>>> val stream = env >>>>>>>> .addSource(new FlinkKafkaConsumer082[String]("location", new >>>>>>>> SimpleStringSchema, prop)) >>>>>>>> >>>>>>>> jar tvf flink-streaming-demo-0.1.jar | grep >>>>>>>> kafka.javaapi.consumer.SimpleConsumer >>>>>>>> >>>>>>>> 5111 Fri Mar 11 14:18:36 UTC 2016 >>>>>>>> *kafka/javaapi/consumer/SimpleConsumer*.class >>>>>>>> >>>>>>>> flink.version = 0.10.2 >>>>>>>> kafka.verison = 0.8.2 >>>>>>>> flink.kafka.connection.verion=0.9.1 >>>>>>>> >>>>>>>> The command that I use to run the flink program in yarn cluster is >>>>>>>> below, >>>>>>>> >>>>>>>> HADOOP_CONF_DIR=/etc/hadoop/conf /usr/share/flink/bin/flink run -c >>>>>>>> com.dataartisans.flink_demo.examples.DriverEventConsumer -m >>>>>>>> yarn-cluster >>>>>>>> -yn 2 /home/balajirajagopalan/flink-streaming-demo-0.1.jar >>>>>>>> >>>>>>>> java.lang.NoClassDefFoundError: >>>>>>>> kafka/javaapi/consumer/SimpleConsumer >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.getPartitionsForTopic(FlinkKafkaConsumer.java:691) >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.<init>(FlinkKafkaConsumer.java:281) >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer082.<init>(FlinkKafkaConsumer082.java:49) >>>>>>>> >>>>>>>> at >>>>>>>> com.dataartisans.flink_demo.examples.DriverEventConsumer$.main(DriverEventConsumer.scala:53) >>>>>>>> >>>>>>>> at >>>>>>>> com.dataartisans.flink_demo.examples.DriverEventConsumer.main(DriverEventConsumer.scala) >>>>>>>> >>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>>>> >>>>>>>> at >>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>>>>>> >>>>>>>> at >>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>> >>>>>>>> at java.lang.reflect.Method.invoke(Method.java:497) >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:497) >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:395) >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.flink.client.program.Client.runBlocking(Client.java:252) >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.flink.client.CliFrontend.executeProgramBlocking(CliFrontend.java:676) >>>>>>>> >>>>>>>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:326) >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:978) >>>>>>>> >>>>>>>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1028) >>>>>>>> >>>>>>>> Caused by: java.lang.ClassNotFoundException: >>>>>>>> kafka.javaapi.consumer.SimpleConsumer >>>>>>>> >>>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >>>>>>>> >>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>>>>>>> >>>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) >>>>>>>> >>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>>>>>>> >>>>>>>> ... 16 more >>>>>>>> >>>>>>>> >>>>>>>> Any help appreciated. >>>>>>>> >>>>>>>> >>>>>>>> balaji >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >