Hi,
flink-connector-kafka_ doesn't exist for 1.0.0. You have to use either
flink-connector-kafka-0.8_ or flink-connector-kafka-0.9_
On Mon, Mar 14, 2016 at 11:17 AM, Balaji Rajagopalan <
balaji.rajagopa...@olacabs.com> wrote:
> What I noticied was that, if I remove the dependency on
> flink-connector-kafka so it is clearly to do something with that
> dependency.
>
>
> On Mon, Mar 14, 2016 at 3:46 PM, Balaji Rajagopalan <
> balaji.rajagopa...@olacabs.com> wrote:
>
>> Robert,
>> I have moved on to latest version of flink of 1.0.0 hoping that will
>> solve my problem with kafka connector . Here is my pom.xml but now I cannot
>> get the code compiled.
>>
>> [ERROR] Failed to execute goal
>> net.alchim31.maven:scala-maven-plugin:3.2.1:compile (scala-compile-first)
>> on project flink-streaming-demo: Execution scala-compile-first of goal
>> net.alchim31.maven:scala-maven-plugin:3.2.1:compile failed: For artifact
>> {null:null:null:jar}: The groupId cannot be empty. -> [Help 1]
>>
>> I read about the above errors in most cases people where able to overcome
>> is by deleting the .m2 directory, and that did not fix the issue for me.
>>
>> What I noticied was that, if I remove the dependency on
>>
>> Here is my pom.xml
>>
>> <?xml version="1.0" encoding="UTF-8"?>
>> <!--
>> Copyright 2015 data Artisans GmbH
>>
>> Licensed under the Apache License, Version 2.0 (the "License");
>> you may not use this file except in compliance with the License.
>> You may obtain a copy of the License at
>>
>> http://www.apache.org/licenses/LICENSE-2.0
>>
>> Unless required by applicable law or agreed to in writing, software
>> distributed under the License is distributed on an "AS IS" BASIS,
>> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>> See the License for the specific language governing permissions and
>> limitations under the License.
>> -->
>> <project xmlns="http://maven.apache.org/POM/4.0.0";
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
>> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
>> http://maven.apache.org/xsd/maven-4.0.0.xsd";>
>> <modelVersion>4.0.0</modelVersion>
>>
>> <groupId>com.dataArtisans</groupId>
>> <artifactId>flink-streaming-demo</artifactId>
>> <version>0.1</version>
>> <packaging>jar</packaging>
>>
>> <name>Flink Streaming Demo</name>
>> <url>http://www.data-artisans.com</url>
>>
>> <properties>
>> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
>> <slf4j.version>1.7.12</slf4j.version>
>> <flink.version>1.0.0</flink.version>
>> <scala.version>2.10</scala.version>
>> </properties>
>>
>> <dependencies>
>>
>>
>>
>> <dependency>
>> <groupId>org.apache.flink</groupId>
>> <artifactId>flink-streaming-scala_${scala.version}</artifactId>
>> <version>${flink.version}</version>
>> </dependency>
>>
>> <dependency>
>> <groupId>org.apache.flink</groupId>
>> <artifactId>flink-runtime-web_${scala.version}</artifactId>
>> <version>${flink.version}</version>
>> </dependency>
>>
>> <dependency>
>> <groupId>org.elasticsearch</groupId>
>> <artifactId>elasticsearch</artifactId>
>> <version>1.7.3</version>
>> <scope>compile</scope>
>> </dependency>
>>
>> <dependency>
>> <groupId>joda-time</groupId>
>> <artifactId>joda-time</artifactId>
>> <version>2.7</version>
>> </dependency>
>>
>> <dependency>
>> <groupId>org.apache.kafka</groupId>
>> <artifactId>kafka_${scala.version}</artifactId>
>> <version>0.8.2.0</version>
>> </dependency>
>>
>> <dependency>
>> <groupId>org.apache.flink</groupId>
>> <artifactId>flink-connector-kafka_${scala.version}</artifactId>
>> <version>${flink.version}</version>
>> </dependency>
>>
>> <dependency>
>> <groupId>org.json4s</groupId>
>> <artifactId>json4s-native_${scala.version}</artifactId>
>> <version>3.3.0</version>
>> </dependency>
>>
>>
>> </dependencies>
>>
>> <build>
>> <plugins>
>>
>> <!-- Scala Compiler -->
>> <plugin>
>> <groupId>net.alchim31.maven</groupId>
>> <artifactId>scala-maven-plugin</artifactId>
>> <version>3.2.1</version>
>> <executions>
>> <!-- Run scala compiler in the process-resources phase, so
>> that dependencies on
>> scala classes can be resolved later in the (Java) compile
>> phase -->
>> <execution>
>> <id>scala-compile-first</id>
>> <phase>process-resources</phase>
>> <goals>
>> <goal>compile</goal>
>> </goals>
>> </execution>
>>
>> <!-- Run scala compiler in the process-test-resources phase,
>> so that dependencies on
>> scala classes can be resolved later in the (Java)
>> test-compile phase -->
>> <execution>
>> <id>scala-test-compile</id>
>> <phase>process-test-resources</phase>
>> <goals>
>> <goal>testCompile</goal>
>> </goals>
>> </execution>
>> </executions>
>> <configuration>
>> <jvmArgs>
>> <jvmArg>-Xms128m</jvmArg>
>> <jvmArg>-Xmx512m</jvmArg>
>> </jvmArgs>
>> </configuration>
>> </plugin>
>>
>> <plugin>
>> <groupId>org.apache.maven.plugins</groupId>
>> <artifactId>maven-dependency-plugin</artifactId>
>> <version>2.9</version>
>> <executions>
>> <execution>
>> <id>unpack</id>
>> <!-- executed just before the package phase -->
>> <phase>prepare-package</phase>
>> <goals>
>> <goal>unpack</goal>
>> </goals>
>> <configuration>
>> <artifactItems>
>> <!-- For Flink connector classes -->
>> <artifactItem>
>> <groupId>org.apache.flink</groupId>
>>
>> <artifactId>flink-connector-kafka_${scala.version}</artifactId>
>> <version>1.0.0</version>
>> <type>jar</type>
>> <overWrite>false</overWrite>
>>
>> <outputDirectory>${project.build.directory}/classes</outputDirectory>
>> <includes>org/apache/flink/**</includes>
>> </artifactItem>
>> <!-- For Kafka API classes -->
>> <artifactItem>
>> <groupId>org.apache.kafka</groupId>
>> <artifactId>kafka_${scala.version}</artifactId>
>> <version>0.8.2.0</version>
>> <type>jar</type>
>> <overWrite>false</overWrite>
>>
>> <outputDirectory>${project.build.directory}/classes</outputDirectory>
>> <includes>kafka/**</includes>
>> </artifactItem>
>> </artifactItems>
>> </configuration>
>> </execution>
>> </executions>
>> </plugin>
>>
>> <!--plugin>
>> <groupId>org.apache.maven.plugins</groupId>
>> <artifactId>maven-jar-plugin</artifactId>
>> <executions>
>>
>> <execution>
>> <id>MBoxParser</id>
>> <phase>package</phase>
>> <goals>
>> <goal>jar</goal>
>> </goals>
>>
>> <configuration>
>> <classifier>MBoxParser</classifier>
>>
>> <archive>
>> <manifestEntries>
>>
>> <main-class>com.dataartisans.flinkTraining.dataSetPreparation.MBoxParser</main-class>
>> </manifestEntries>
>> </archive>
>>
>> <includes>
>> <include>**/MBoxParser.class</include>
>> <include>**/MBoxParser$*.class</include>
>> </includes>
>> </configuration>
>> </execution>
>>
>> </executions>
>> </plugin-->
>>
>> <plugin>
>> <groupId>org.apache.maven.plugins</groupId>
>> <artifactId>maven-compiler-plugin</artifactId>
>> <version>3.1</version>
>> <configuration>
>> <source>1.8</source> <!-- If you want to use Java 8, change
>> this to "1.8" -->
>> <target>1.8</target> <!-- If you want to use Java 8, change
>> this to "1.8" -->
>> </configuration>
>> </plugin>
>>
>> <plugin>
>> <groupId>org.apache.rat</groupId>
>> <artifactId>apache-rat-plugin</artifactId>
>> <version>0.10</version><!--$NO-MVN-MAN-VER$-->
>> <inherited>false</inherited>
>> <executions>
>> <execution>
>> <phase>verify</phase>
>> <goals>
>> <goal>check</goal>
>> </goals>
>> </execution>
>> </executions>
>> <configuration>
>> <excludeSubProjects>false</excludeSubProjects>
>> <numUnapprovedLicenses>0</numUnapprovedLicenses>
>> <licenses>
>> <!-- Enforce this license:
>> Copyright 2015 data Artisans GmbH
>>
>> Licensed under the Apache License, Version 2.0 (the
>> "License");
>> you may not use this file except in compliance with the
>> License.
>> You may obtain a copy of the License at
>>
>> http://www.apache.org/licenses/LICENSE-2.0
>>
>> Unless required by applicable law or agreed to in
>> writing, software
>> distributed under the License is distributed on an "AS
>> IS" BASIS,
>> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
>> express or implied.
>> See the License for the specific language governing
>> permissions and
>> limitations under the License.
>> -->
>> <license
>> implementation="org.apache.rat.analysis.license.SimplePatternBasedLicense">
>> <licenseFamilyCategory>AL2 </licenseFamilyCategory>
>> <licenseFamilyName>Apache License
>> 2.0</licenseFamilyName>
>> <notes />
>> <patterns>
>> <pattern>Copyright 2015 data Artisans GmbH</pattern>
>> <pattern>Licensed under the Apache License, Version
>> 2.0 (the "License");</pattern>
>> </patterns>
>> </license>
>> </licenses>
>> <licenseFamilies>
>> <licenseFamily
>> implementation="org.apache.rat.license.SimpleLicenseFamily">
>> <familyName>Apache License 2.0</familyName>
>> </licenseFamily>
>> </licenseFamilies>
>> <excludes>
>> <!-- Additional files like .gitignore etc.-->
>> <exclude>**/.*</exclude>
>> <exclude>**/*.prefs</exclude>
>> <exclude>**/*.properties</exclude>
>> <exclude>**/*.log</exclude>
>> <exclude>*.txt/**</exclude>
>> <!-- Administrative files in the main trunk. -->
>> <exclude>**/README.md</exclude>
>> <exclude>CHANGELOG</exclude>
>> <!-- Build files -->
>> <exclude>**/*.iml</exclude>
>> <!-- Generated content -->
>> <exclude>**/target/**</exclude>
>> <exclude>**/build/**</exclude>
>> </excludes>
>> </configuration>
>> </plugin>
>>
>> <plugin>
>> <groupId>org.apache.maven.plugins</groupId>
>> <artifactId>maven-checkstyle-plugin</artifactId>
>> <version>2.12.1</version>
>> <executions>
>> <execution>
>> <id>validate</id>
>> <phase>validate</phase>
>> <goals>
>> <goal>check</goal>
>> </goals>
>> </execution>
>> </executions>
>> <configuration>
>> <configLocation>/tools/maven/checkstyle.xml</configLocation>
>> <logViolationsToConsole>true</logViolationsToConsole>
>> </configuration>
>> </plugin>
>>
>> </plugins>
>>
>> </build>
>> </project>
>>
>>
>> On Mon, Mar 14, 2016 at 3:15 PM, Robert Metzger <rmetz...@apache.org>
>> wrote:
>>
>>> Can you send me the full build file to further investigate the issue?
>>>
>>> On Fri, Mar 11, 2016 at 4:56 PM, Balaji Rajagopalan <
>>> balaji.rajagopa...@olacabs.com> wrote:
>>>
>>>> Robert,
>>>> That did not fix it ( using flink and connector same version) . Tried
>>>> with scala version 2.11, so will try to see scala 2.10 makes any
>>>> difference.
>>>>
>>>> balaji
>>>>
>>>> On Fri, Mar 11, 2016 at 8:06 PM, Robert Metzger <rmetz...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> you have to use the same version for all dependencies from the
>>>>> "org.apache.flink" group.
>>>>>
>>>>> You said these are the versions you are using:
>>>>>
>>>>> flink.version = 0.10.2
>>>>> kafka.verison = 0.8.2
>>>>> flink.kafka.connection.verion=0.9.1
>>>>>
>>>>> For the connector, you also need to use 0.10.2.
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Mar 11, 2016 at 9:56 AM, Balaji Rajagopalan <
>>>>> balaji.rajagopa...@olacabs.com> wrote:
>>>>>
>>>>>> I am tyring to use the flink kafka connector, for this I have
>>>>>> specified the kafka connector dependency and created a fat jar since
>>>>>> default flink installation does not contain kafka connector jars. I have
>>>>>> made sure that flink-streaming-demo-0.1.jar has the
>>>>>> kafka.javaapi.consumer.SimpleConsumer.class but still I see the class not
>>>>>> found exception.
>>>>>>
>>>>>> The code for kafka connector in flink.
>>>>>>
>>>>>> val env = StreamExecutionEnvironment.getExecutionEnvironment
>>>>>> val prop:Properties = new Properties()
>>>>>> prop.setProperty("zookeeper.connect","somezookeer:2181")
>>>>>> prop.setProperty("group.id","some-group")
>>>>>> prop.setProperty("bootstrap.servers","somebroker:9092")
>>>>>>
>>>>>> val stream = env
>>>>>> .addSource(new FlinkKafkaConsumer082[String]("location", new
>>>>>> SimpleStringSchema, prop))
>>>>>>
>>>>>> jar tvf flink-streaming-demo-0.1.jar | grep
>>>>>> kafka.javaapi.consumer.SimpleConsumer
>>>>>>
>>>>>> 5111 Fri Mar 11 14:18:36 UTC 2016
>>>>>> *kafka/javaapi/consumer/SimpleConsumer*.class
>>>>>>
>>>>>> flink.version = 0.10.2
>>>>>> kafka.verison = 0.8.2
>>>>>> flink.kafka.connection.verion=0.9.1
>>>>>>
>>>>>> The command that I use to run the flink program in yarn cluster is
>>>>>> below,
>>>>>>
>>>>>> HADOOP_CONF_DIR=/etc/hadoop/conf /usr/share/flink/bin/flink run -c
>>>>>> com.dataartisans.flink_demo.examples.DriverEventConsumer -m yarn-cluster
>>>>>> -yn 2 /home/balajirajagopalan/flink-streaming-demo-0.1.jar
>>>>>>
>>>>>> java.lang.NoClassDefFoundError: kafka/javaapi/consumer/SimpleConsumer
>>>>>>
>>>>>> at
>>>>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.getPartitionsForTopic(FlinkKafkaConsumer.java:691)
>>>>>>
>>>>>> at
>>>>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.<init>(FlinkKafkaConsumer.java:281)
>>>>>>
>>>>>> at
>>>>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer082.<init>(FlinkKafkaConsumer082.java:49)
>>>>>>
>>>>>> at
>>>>>> com.dataartisans.flink_demo.examples.DriverEventConsumer$.main(DriverEventConsumer.scala:53)
>>>>>>
>>>>>> at
>>>>>> com.dataartisans.flink_demo.examples.DriverEventConsumer.main(DriverEventConsumer.scala)
>>>>>>
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>
>>>>>> at
>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>>
>>>>>> at
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>
>>>>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>>>>>
>>>>>> at
>>>>>> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:497)
>>>>>>
>>>>>> at
>>>>>> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:395)
>>>>>>
>>>>>> at org.apache.flink.client.program.Client.runBlocking(Client.java:252)
>>>>>>
>>>>>> at
>>>>>> org.apache.flink.client.CliFrontend.executeProgramBlocking(CliFrontend.java:676)
>>>>>>
>>>>>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:326)
>>>>>>
>>>>>> at
>>>>>> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:978)
>>>>>>
>>>>>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1028)
>>>>>>
>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>> kafka.javaapi.consumer.SimpleConsumer
>>>>>>
>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>>>>>>
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>>
>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>>>>>>
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>
>>>>>> ... 16 more
>>>>>>
>>>>>>
>>>>>> Any help appreciated.
>>>>>>
>>>>>>
>>>>>> balaji
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
