Re: Contributing to Flink
Hi Niraj, Welcome to the Flink community ;) I'm really excited that you want to contribute to our project, and since you've asked for something in the security area, I actually have something very concrete in mind. We recently added some support for accessing (Kerberos) secured HDFS clusters in Flink: https://issues.apache.org/jira/browse/FLINK-1504. However, the implementation is very simple because it assumes that every worker of Flink (TaskManager) is authenticated with Kerberos (kinit). Its not very practical for large setups because you have to ssh to all machines to log into Kerberos. What I would really like to have in Flink would be an way to transfer the authentication tokens form the JobManager (master) to the TaskManagers. This way, users only have to be authenticated with Kerberos at the JobManager, and Flink is taking care of the rest. As far as I understood it, Hadoop has already all the utilities in place for getting and transferring the delegation tokens. Max Michels, another committer in our project has quite a good understanding of the details there. It would be great if you (Max) could chime in if I forgot something. If you are interested in working on this, you can file a JIRA ( https://issues.apache.org/jira/browse/FLINK) for tracking the progress and discussing the details. If not I'm sure we'll come up with more interesting ideas. Robert On Thu, Feb 26, 2015 at 11:07 PM, Henry Saputra henry.sapu...@gmail.com wrote: Hi Niraj, Thanks for your interest at Apache Flink. The quickest is to just give Flink a spin and figure out how it works. This would get you start on how it works before actually doing work on Flink =) Please do visit Flink how to contribute page [1] and subscribe to dev mailing list [2] to start following up. Welcome =) [1] http://flink.apache.org/how-to-contribute.html [2] http://flink.apache.org/community.html#mailing-lists On Thu, Feb 26, 2015 at 1:45 PM, Rai, Niraj niraj@intel.com wrote: Hi Flink Dev, I am looking to contribute to Flink, especially in the area of security. In the past, I have contributed to Pig, Hive and HDFS. I would really appreciate, if I can get some work assigned to me. Looking forward to hear back from the development community of Flink. Thanks Niraj
Re: gelli graph algorithm
Hello, Also, for guidelines on how to implement a graph algorithm in Gelly, you can use the provided examples: https://github.com/apache/flink/tree/master/flink-staging/flink-gelly/src/main/java/org/apache/flink/graph/example Have fun! Andra On Thu, Feb 26, 2015 at 5:31 PM, Fabian Hueske fhue...@gmail.com wrote: Hi Martin, as a start, there is a PR with Gelly documentation: https://github.com/vasia/flink/blob/gelly-guide/docs/gelly_guide.md Cheers, Fabian 2015-02-26 17:12 GMT+01:00 Martin Neumann mneum...@spotify.com: Hej, I was busy with other stuff for a while but I hope I will have more time to work on Flink and Graphs again now. I need to do some basic analytic's on a large graph set (stuff like degree distribution, triangle count, component size distribution etc.) Is there anything implemented in Gelli already that I can use? Is there some basic documentation about Gelli somewhere? If I need to program those algorithm myself I could do it in a way that we can integrate them in Gelli. But I might need some help to make sure I design them in a way that fits, so I might need some guidance/help. cheers Martin
gelli graph algorithm
Hej, I was busy with other stuff for a while but I hope I will have more time to work on Flink and Graphs again now. I need to do some basic analytic's on a large graph set (stuff like degree distribution, triangle count, component size distribution etc.) Is there anything implemented in Gelli already that I can use? Is there some basic documentation about Gelli somewhere? If I need to program those algorithm myself I could do it in a way that we can integrate them in Gelli. But I might need some help to make sure I design them in a way that fits, so I might need some guidance/help. cheers Martin
gelli graph algorithm
Hej, I was busy with other stuff for a while but I hope I will have more time to work on Flink and Graphs again now. I need to do some basic analytic's on a large graph set (stuff like degree distribution, triangle count, component size distribution etc.) Is there anything implemented in Gelli already that I can use? Is there some basic documentation about Gelli somewhere? If I need to program those algorithm myself I could do it in a way that we can integrate them in Gelli. But I might need some help to make sure I design them in a way that fits, so I might need some guidance/help. cheers Martin
Re: gelli graph algorithm
Hi Martin, as a start, there is a PR with Gelly documentation: https://github.com/vasia/flink/blob/gelly-guide/docs/gelly_guide.md Cheers, Fabian 2015-02-26 17:12 GMT+01:00 Martin Neumann mneum...@spotify.com: Hej, I was busy with other stuff for a while but I hope I will have more time to work on Flink and Graphs again now. I need to do some basic analytic's on a large graph set (stuff like degree distribution, triangle count, component size distribution etc.) Is there anything implemented in Gelli already that I can use? Is there some basic documentation about Gelli somewhere? If I need to program those algorithm myself I could do it in a way that we can integrate them in Gelli. But I might need some help to make sure I design them in a way that fits, so I might need some guidance/help. cheers Martin
Re: Could not build up connection to JobManager
Hi, It’s great to help out. :) Setting 127.0.0.1 instead of “localhost” in jobmanager.rpc.address, helped to build the connection to the jobmanager. Apparently localhost resolving is different in webclient and the jobmanager. I think it’s good to set jobmanager.rpc.address: 127.0.0.1 in future builds. But then I get this error when I tried to run examples. I don’t know if I should move this issue to another thread. If so please tell me. bin/flink run /Users/Vidura/Documents/Development/flink/flink-dist/target/flink-0.9-SNAPSHOT-bin/flink-0.9-SNAPSHOT/examples/flink-java-examples-0.9-SNAPSHOT-WordCount.jar /Users/Vidura/Documents/Development/flink/flink-dist/target/flink-0.9-SNAPSHOT-bin/flink-0.9-SNAPSHOT/hamlet.txt $FLINK_DIRECTORY/count 20:46:21,998 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 02/26/2015 20:46:23 Job execution switched to status RUNNING. 02/26/2015 20:46:23 CHAIN DataSource (at getTextDataSet(WordCount.java:141) (org.apache.flink.api.java.io.TextInputFormat)) - FlatMap (FlatMap at main(WordCount.java:69)) - Combine(SUM(1), at main(WordCount.java:72)(1/1) switched to SCHEDULED 02/26/2015 20:46:23 CHAIN DataSource (at getTextDataSet(WordCount.java:141) (org.apache.flink.api.java.io.TextInputFormat)) - FlatMap (FlatMap at main(WordCount.java:69)) - Combine(SUM(1), at main(WordCount.java:72)(1/1) switched to DEPLOYING 02/26/2015 20:48:03 CHAIN DataSource (at getTextDataSet(WordCount.java:141) (org.apache.flink.api.java.io.TextInputFormat)) - FlatMap (FlatMap at main(WordCount.java:69)) - Combine(SUM(1), at main(WordCount.java:72)(1/1) switched to FAILED akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/taskmanager#-1628133761]] after [10 ms] at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:333) at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117) at scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:694) at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:691) at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467) at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419) at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423) at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375) at java.lang.Thread.run(Thread.java:745) 02/26/2015 20:48:03 Job execution switched to status FAILING. 02/26/2015 20:48:03 Reduce (SUM(1), at main(WordCount.java:72)(1/1) switched to CANCELED 02/26/2015 20:48:03 DataSink(CsvOutputFormat (path: /Users/Vidura/Documents/Development/flink/flink-dist/target/flink-0.9-SNAPSHOT-bin/flink-0.9-SNAPSHOT/count, delimiter: ))(1/1) switched to CANCELED 02/26/2015 20:48:03 Job execution switched to status FAILED. org.apache.flink.client.program.ProgramInvocationException: The program execution failed. at org.apache.flink.client.program.Client.run(Client.java:344) at org.apache.flink.client.program.Client.run(Client.java:306) at org.apache.flink.client.program.Client.run(Client.java:300) at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:55) at org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:82) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353) at org.apache.flink.client.program.Client.run(Client.java:250) at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:371) at org.apache.flink.client.CliFrontend.run(CliFrontend.java:344) at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1087) at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1114) Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed. at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$receiveWithLogMessages$1.applyOrElse(JobManager.scala:284) at scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33) at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33) at
Re: Log4j configuration JUnit vs. Scalatest in IntelliJ
On 25 Feb 2015, at 16:35, Till Rohrmann trohrm...@apache.org wrote: The reason for this behaviour is the following: The log4j-test.properties is not a standard log4j properties file. It is only used if it is explicitly given to the executing JVM by -Dlog4j.configuration. The parent pom defines for the surefire plugin a corresponding argument line with this option. Intellij uses these maven settings for executing Java JUnit tests. For the scala tests which are executed with JUnitRunner, these settings are not used. I don't know why this is the case for the scala tests. But in order to get consistent behaviour you can turn off the maven settings import by unticking Build, Execution, Deployment - Build Tools - Maven - Running Tests - argLine in the IntelliJ settings. Thanks. Makes more sense now. :-)
[jira] [Created] (FLINK-1613) Cannost submit to remote ExecutionEnvironment from IDE
Alexander Alexandrov created FLINK-1613:
---
Summary: Cannost submit to remote ExecutionEnvironment from IDE
Key: FLINK-1613
URL: https://issues.apache.org/jira/browse/FLINK-1613
Project: Flink
Issue Type: Bug
Components: Distributed Runtime
Affects Versions: 0.8.1
Environment: * Ubuntu Linux 14.04
* Flink 0.9-SNAPSHOT or 0.8.1 running in standalone mode on localhost
Reporter: Alexander Alexandrov
Fix For: 0.9, 0.8.2
I am reporting this as [~rmetzler] mentioned offline that it was working in the
past.
At the moment it is not possible to submit jobs directly from the IDE. Both the
Java and the Scala quickstart guides fail on both 0.8.1 and 0.9-SNAPSHOT with
ClassNotFoundException exceptions.
To reproduce the error, run the quickstart scripts and change the
ExecutionEnvironment initialization:
{code:java}
env = ExecutionEnvironment.createRemoteEnvironment(localhost, 6123)
{code}
This is the cause for Java:
{noformat}
Caused by: java.lang.ClassNotFoundException:
org.myorg.quickstart.WordCount$LineSplitter
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:274)
at
org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:54)
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:274)
at
org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:236)
at
org.apache.flink.runtime.operators.util.TaskConfig.getStubWrapper(TaskConfig.java:281)
{noformat}
This is for Scala:
{noformat}
java.lang.ClassNotFoundException: org.myorg.quickstart.WordCount$$anon$2$$anon$1
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:274)
at
org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:54)
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:274)
at
org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:236)
at
org.apache.flink.api.java.typeutils.runtime.RuntimeSerializerFactory.readParametersFromConfig(RuntimeSerializerFactory.java:76)
at
org.apache.flink.runtime.operators.util.TaskConfig.getTypeSerializerFactory(TaskConfig.java:1084)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Could not build up connection to JobManager
Hi Dulaj! Thanks for helping to debug. My guess is that you are seeing now the same thing between JobManager and TaskManager as you saw before between JobManager and JobClient. I have a patch pending that should help the issue (see https://issues.apache.org/jira/browse/FLINK-1608), let's see if that solves it. What seems not right is that the JobManager initially accepted the TaskManager and later the communication. Can you paste the TaskManager log as well? Also: There must be something fairly unique about your network configuration, as it works on all other setups that we use (locally, cloud, test servers, YARN, ...). Can you paste your ipconfig / ifconfig by any chance? Greetings, Stephan On Thu, Feb 26, 2015 at 4:33 PM, Dulaj Viduranga vidura...@icloud.com wrote: Hi, It’s great to help out. :) Setting 127.0.0.1 instead of “localhost” in jobmanager.rpc.address, helped to build the connection to the jobmanager. Apparently localhost resolving is different in webclient and the jobmanager. I think it’s good to set jobmanager.rpc.address: 127.0.0.1 in future builds. But then I get this error when I tried to run examples. I don’t know if I should move this issue to another thread. If so please tell me. bin/flink run /Users/Vidura/Documents/Development/flink/flink-dist/target/flink-0.9-SNAPSHOT-bin/flink-0.9-SNAPSHOT/examples/flink-java-examples-0.9-SNAPSHOT-WordCount.jar /Users/Vidura/Documents/Development/flink/flink-dist/target/flink-0.9-SNAPSHOT-bin/flink-0.9-SNAPSHOT/hamlet.txt $FLINK_DIRECTORY/count 20:46:21,998 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 02/26/2015 20:46:23 Job execution switched to status RUNNING. 02/26/2015 20:46:23 CHAIN DataSource (at getTextDataSet(WordCount.java:141) (org.apache.flink.api.java.io.TextInputFormat)) - FlatMap (FlatMap at main(WordCount.java:69)) - Combine(SUM(1), at main(WordCount.java:72)(1/1) switched to SCHEDULED 02/26/2015 20:46:23 CHAIN DataSource (at getTextDataSet(WordCount.java:141) (org.apache.flink.api.java.io.TextInputFormat)) - FlatMap (FlatMap at main(WordCount.java:69)) - Combine(SUM(1), at main(WordCount.java:72)(1/1) switched to DEPLOYING 02/26/2015 20:48:03 CHAIN DataSource (at getTextDataSet(WordCount.java:141) (org.apache.flink.api.java.io.TextInputFormat)) - FlatMap (FlatMap at main(WordCount.java:69)) - Combine(SUM(1), at main(WordCount.java:72)(1/1) switched to FAILED akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/taskmanager#-1628133761]] after [10 ms] at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:333) at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117) at scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:694) at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:691) at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467) at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419) at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423) at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375) at java.lang.Thread.run(Thread.java:745) 02/26/2015 20:48:03 Job execution switched to status FAILING. 02/26/2015 20:48:03 Reduce (SUM(1), at main(WordCount.java:72)(1/1) switched to CANCELED 02/26/2015 20:48:03 DataSink(CsvOutputFormat (path: /Users/Vidura/Documents/Development/flink/flink-dist/target/flink-0.9-SNAPSHOT-bin/flink-0.9-SNAPSHOT/count, delimiter: ))(1/1) switched to CANCELED 02/26/2015 20:48:03 Job execution switched to status FAILED. org.apache.flink.client.program.ProgramInvocationException: The program execution failed. at org.apache.flink.client.program.Client.run(Client.java:344) at org.apache.flink.client.program.Client.run(Client.java:306) at org.apache.flink.client.program.Client.run(Client.java:300) at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:55) at org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:82) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353) at
Contributing to Flink
Hi Flink Dev, I am looking to contribute to Flink, especially in the area of security. In the past, I have contributed to Pig, Hive and HDFS. I would really appreciate, if I can get some work assigned to me. Looking forward to hear back from the development community of Flink. Thanks Niraj
Re: Questions about flink-streaming-examples
Thanks for clarifying Marton!
I was on the latest build already. However, my local maven repository
contained old jars. After removing all flink-jars from my local maven
repository it works!
Why does maven no automatically update the local repository?
-Matthias
On 02/26/2015 09:20 AM, Márton Balassi wrote:
Dear Mathias,
Thanks for reporting the issue. I have successfully built
flink-streaming-examples with maven, you can depend on test classes, the
following in the pom does the trick:
dependency
groupIdorg.apache.flink/groupId
artifactIdflink-streaming-core/artifactId
version${project.version}/version
scopetest/scope
classifiertests/classifier
/dependency
This tells maven that the test classes of the examples (scope) might depend
on the tests of the core (classifier). Without explicitly stating the tests
classifier indeed maven only uses the classes under src/main.
I think the problem occurred because you only built
flink-streaming-examples and you might have used an outdated version of the
flink-streaming-core as a dependency installed to your local maven
repository - which did not include the StreamingTestBase yet. The change
that we are talking is about a week old, I wrote it.
Kind regards,
Marton
On Thu, Feb 26, 2015 at 9:04 AM, Matthias J. Sax
mj...@informatik.hu-berlin.de wrote:
Hi,
I just build flink-streaming and avoid the problem. I guess, that the
issue is related to the module structure and dependencies.
flink-streaming-examples uses
org.apache.flink.streaming.util.StreamingProgramTestBase (that is
defined in flink-streaming-core/src/TEST
From my understanding, maven does not support dependencies into src/test
but only into src/main (maybe I am wrong about this).
I would suggest, to move StreamingProgramTestBase into a new module
flink-streaming-testUtils (into src/main) and change the dependency of
flink-streaming-examples accordingly.
Why the build work on the parent-project, is still a riddle to me... But
it seems that the dependency into scr/test can be resolved for this case.
-Matthias
On 02/25/2015 06:12 PM, Dulaj Viduranga wrote:
Hi,
Were you able to solve this? It seams the examples depend on
flink-streaming-core”. I think you have to build it before.
On Feb 25, 2015, at 10:04 PM, Matthias J. Sax
mj...@informatik.hu-berlin.de wrote:
Thanks!
Even if I am not a building and/or maven expert, it seems to me that the
dependencies are not configures correctly... No clue how to fix it. But
it might be a good idea to have a look into it IMHO.
-Matthias
On 02/25/2015 05:07 PM, Stephan Ewen wrote:
Hi Matthias!
Can you try mvn clean package? It package is generally preferable to
compile, in my opinion.
There may be a dependency through a test jar that is not properly
handled
when you do not execute the package goal.
Stephan
Am 25.02.2015 16:54 schrieb Max Michels m...@apache.org:
Hi Matthias,
Did you really pull from the latest master? I just tried to compile
flink-streaming-examples using mvn clean compile test-compile and it
worked.
Best regards,
Max
On Wed, Feb 25, 2015 at 4:13 PM, Matthias J. Sax
mj...@informatik.hu-berlin.de wrote:
Hi,
I tried to build flink-streaming-examples, but got an compilation
error.
If I build flink-streaming (that implicitly builds
flink-streaming-examples), it works.
I tried it on a fresh clone of flink/master using mvn clean compile
test-compile
Can anybody explain this behavior?
-Matthias
mjsax@T420s-dbis-mjsax
:~/workspace_flink/dummy/flink/flink-staging/flink-streaming/flink-streaming-examples$
mvn clean compile test-compile
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the
effective
model for org.apache.flink:flink-streaming-examples:jar:0.9-SNAPSHOT
[WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique
but found duplicate declaration of plugin
org.apache.maven.plugins:maven-jar-plugin @ line 462, column 12
[WARNING]
[WARNING] It is highly recommended to fix these problems because
they
threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer
support building such malformed projects.
[WARNING]
[INFO]
[INFO]
[INFO] Building flink-streaming-examples 0.9-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @
flink-streaming-examples ---
[INFO] Deleting
/home/mjsax/workspace_flink/dummy/flink/flink-staging/flink-streaming/flink-streaming-examples/target
[INFO]
[INFO] --- maven-checkstyle-plugin:2.12.1:check (validate) @
flink-streaming-examples ---
[INFO]
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-maven) @
flink-streaming-examples ---
[INFO]
[INFO] ---
Re: Questions about flink-streaming-examples
To update the local repository, you have to do execute the install goal.
I can recommend to always do a mvn clean install
On Thu, Feb 26, 2015 at 10:11 AM, Matthias J. Sax
mj...@informatik.hu-berlin.de wrote:
Thanks for clarifying Marton!
I was on the latest build already. However, my local maven repository
contained old jars. After removing all flink-jars from my local maven
repository it works!
Why does maven no automatically update the local repository?
-Matthias
On 02/26/2015 09:20 AM, Márton Balassi wrote:
Dear Mathias,
Thanks for reporting the issue. I have successfully built
flink-streaming-examples with maven, you can depend on test classes, the
following in the pom does the trick:
dependency
groupIdorg.apache.flink/groupId
artifactIdflink-streaming-core/artifactId
version${project.version}/version
scopetest/scope
classifiertests/classifier
/dependency
This tells maven that the test classes of the examples (scope) might
depend
on the tests of the core (classifier). Without explicitly stating the
tests
classifier indeed maven only uses the classes under src/main.
I think the problem occurred because you only built
flink-streaming-examples and you might have used an outdated version of
the
flink-streaming-core as a dependency installed to your local maven
repository - which did not include the StreamingTestBase yet. The change
that we are talking is about a week old, I wrote it.
Kind regards,
Marton
On Thu, Feb 26, 2015 at 9:04 AM, Matthias J. Sax
mj...@informatik.hu-berlin.de wrote:
Hi,
I just build flink-streaming and avoid the problem. I guess, that the
issue is related to the module structure and dependencies.
flink-streaming-examples uses
org.apache.flink.streaming.util.StreamingProgramTestBase (that is
defined in flink-streaming-core/src/TEST
From my understanding, maven does not support dependencies into src/test
but only into src/main (maybe I am wrong about this).
I would suggest, to move StreamingProgramTestBase into a new module
flink-streaming-testUtils (into src/main) and change the dependency of
flink-streaming-examples accordingly.
Why the build work on the parent-project, is still a riddle to me... But
it seems that the dependency into scr/test can be resolved for this
case.
-Matthias
On 02/25/2015 06:12 PM, Dulaj Viduranga wrote:
Hi,
Were you able to solve this? It seams the examples depend on
flink-streaming-core”. I think you have to build it before.
On Feb 25, 2015, at 10:04 PM, Matthias J. Sax
mj...@informatik.hu-berlin.de wrote:
Thanks!
Even if I am not a building and/or maven expert, it seems to me that
the
dependencies are not configures correctly... No clue how to fix it.
But
it might be a good idea to have a look into it IMHO.
-Matthias
On 02/25/2015 05:07 PM, Stephan Ewen wrote:
Hi Matthias!
Can you try mvn clean package? It package is generally preferable
to
compile, in my opinion.
There may be a dependency through a test jar that is not properly
handled
when you do not execute the package goal.
Stephan
Am 25.02.2015 16:54 schrieb Max Michels m...@apache.org:
Hi Matthias,
Did you really pull from the latest master? I just tried to compile
flink-streaming-examples using mvn clean compile test-compile and
it
worked.
Best regards,
Max
On Wed, Feb 25, 2015 at 4:13 PM, Matthias J. Sax
mj...@informatik.hu-berlin.de wrote:
Hi,
I tried to build flink-streaming-examples, but got an compilation
error.
If I build flink-streaming (that implicitly builds
flink-streaming-examples), it works.
I tried it on a fresh clone of flink/master using mvn clean
compile
test-compile
Can anybody explain this behavior?
-Matthias
mjsax@T420s-dbis-mjsax
:~/workspace_flink/dummy/flink/flink-staging/flink-streaming/flink-streaming-examples$
mvn clean compile test-compile
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the
effective
model for org.apache.flink:flink-streaming-examples:jar:0.9-SNAPSHOT
[WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be
unique
but found duplicate declaration of plugin
org.apache.maven.plugins:maven-jar-plugin @ line 462, column 12
[WARNING]
[WARNING] It is highly recommended to fix these problems because
they
threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer
support building such malformed projects.
[WARNING]
[INFO]
[INFO]
[INFO] Building flink-streaming-examples 0.9-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @
flink-streaming-examples ---
[INFO]
Re: Questions about flink-streaming-examples
If the streaming-examples module uses the classifier tag to add the
test-core dependency then we should change it into type tag as
recommended by maven [1]. Otherwise it might come to build failures if the
install lifecycle is not executed.
The dependency import should look like:
dependency
groupIdorg.apache.flink/groupId
artifactIdflink-streaming-core/artifactId
version${project.version}/version
scopetest/scope
typetest-jar/type
/dependency
[1] http://maven.apache.org/guides/mini/guide-attached-tests.html
On Thu, Feb 26, 2015 at 10:31 AM, Robert Metzger rmetz...@apache.org
wrote:
To update the local repository, you have to do execute the install goal.
I can recommend to always do a mvn clean install
On Thu, Feb 26, 2015 at 10:11 AM, Matthias J. Sax
mj...@informatik.hu-berlin.de wrote:
Thanks for clarifying Marton!
I was on the latest build already. However, my local maven repository
contained old jars. After removing all flink-jars from my local maven
repository it works!
Why does maven no automatically update the local repository?
-Matthias
On 02/26/2015 09:20 AM, Márton Balassi wrote:
Dear Mathias,
Thanks for reporting the issue. I have successfully built
flink-streaming-examples with maven, you can depend on test classes,
the
following in the pom does the trick:
dependency
groupIdorg.apache.flink/groupId
artifactIdflink-streaming-core/artifactId
version${project.version}/version
scopetest/scope
classifiertests/classifier
/dependency
This tells maven that the test classes of the examples (scope) might
depend
on the tests of the core (classifier). Without explicitly stating the
tests
classifier indeed maven only uses the classes under src/main.
I think the problem occurred because you only built
flink-streaming-examples and you might have used an outdated version of
the
flink-streaming-core as a dependency installed to your local maven
repository - which did not include the StreamingTestBase yet. The
change
that we are talking is about a week old, I wrote it.
Kind regards,
Marton
On Thu, Feb 26, 2015 at 9:04 AM, Matthias J. Sax
mj...@informatik.hu-berlin.de wrote:
Hi,
I just build flink-streaming and avoid the problem. I guess, that
the
issue is related to the module structure and dependencies.
flink-streaming-examples uses
org.apache.flink.streaming.util.StreamingProgramTestBase (that is
defined in flink-streaming-core/src/TEST
From my understanding, maven does not support dependencies into
src/test
but only into src/main (maybe I am wrong about this).
I would suggest, to move StreamingProgramTestBase into a new module
flink-streaming-testUtils (into src/main) and change the dependency
of
flink-streaming-examples accordingly.
Why the build work on the parent-project, is still a riddle to me...
But
it seems that the dependency into scr/test can be resolved for this
case.
-Matthias
On 02/25/2015 06:12 PM, Dulaj Viduranga wrote:
Hi,
Were you able to solve this? It seams the examples depend on
flink-streaming-core”. I think you have to build it before.
On Feb 25, 2015, at 10:04 PM, Matthias J. Sax
mj...@informatik.hu-berlin.de wrote:
Thanks!
Even if I am not a building and/or maven expert, it seems to me that
the
dependencies are not configures correctly... No clue how to fix it.
But
it might be a good idea to have a look into it IMHO.
-Matthias
On 02/25/2015 05:07 PM, Stephan Ewen wrote:
Hi Matthias!
Can you try mvn clean package? It package is generally preferable
to
compile, in my opinion.
There may be a dependency through a test jar that is not properly
handled
when you do not execute the package goal.
Stephan
Am 25.02.2015 16:54 schrieb Max Michels m...@apache.org:
Hi Matthias,
Did you really pull from the latest master? I just tried to
compile
flink-streaming-examples using mvn clean compile test-compile
and
it
worked.
Best regards,
Max
On Wed, Feb 25, 2015 at 4:13 PM, Matthias J. Sax
mj...@informatik.hu-berlin.de wrote:
Hi,
I tried to build flink-streaming-examples, but got an compilation
error.
If I build flink-streaming (that implicitly builds
flink-streaming-examples), it works.
I tried it on a fresh clone of flink/master using mvn clean
compile
test-compile
Can anybody explain this behavior?
-Matthias
mjsax@T420s-dbis-mjsax
:~/workspace_flink/dummy/flink/flink-staging/flink-streaming/flink-streaming-examples$
mvn clean compile test-compile
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the
effective
model for
org.apache.flink:flink-streaming-examples:jar:0.9-SNAPSHOT
[WARNING]
Re: Questions about flink-streaming-examples
Dear Mathias,
Thanks for reporting the issue. I have successfully built
flink-streaming-examples with maven, you can depend on test classes, the
following in the pom does the trick:
dependency
groupIdorg.apache.flink/groupId
artifactIdflink-streaming-core/artifactId
version${project.version}/version
scopetest/scope
classifiertests/classifier
/dependency
This tells maven that the test classes of the examples (scope) might depend
on the tests of the core (classifier). Without explicitly stating the tests
classifier indeed maven only uses the classes under src/main.
I think the problem occurred because you only built
flink-streaming-examples and you might have used an outdated version of the
flink-streaming-core as a dependency installed to your local maven
repository - which did not include the StreamingTestBase yet. The change
that we are talking is about a week old, I wrote it.
Kind regards,
Marton
On Thu, Feb 26, 2015 at 9:04 AM, Matthias J. Sax
mj...@informatik.hu-berlin.de wrote:
Hi,
I just build flink-streaming and avoid the problem. I guess, that the
issue is related to the module structure and dependencies.
flink-streaming-examples uses
org.apache.flink.streaming.util.StreamingProgramTestBase (that is
defined in flink-streaming-core/src/TEST
From my understanding, maven does not support dependencies into src/test
but only into src/main (maybe I am wrong about this).
I would suggest, to move StreamingProgramTestBase into a new module
flink-streaming-testUtils (into src/main) and change the dependency of
flink-streaming-examples accordingly.
Why the build work on the parent-project, is still a riddle to me... But
it seems that the dependency into scr/test can be resolved for this case.
-Matthias
On 02/25/2015 06:12 PM, Dulaj Viduranga wrote:
Hi,
Were you able to solve this? It seams the examples depend on
flink-streaming-core”. I think you have to build it before.
On Feb 25, 2015, at 10:04 PM, Matthias J. Sax
mj...@informatik.hu-berlin.de wrote:
Thanks!
Even if I am not a building and/or maven expert, it seems to me that the
dependencies are not configures correctly... No clue how to fix it. But
it might be a good idea to have a look into it IMHO.
-Matthias
On 02/25/2015 05:07 PM, Stephan Ewen wrote:
Hi Matthias!
Can you try mvn clean package? It package is generally preferable to
compile, in my opinion.
There may be a dependency through a test jar that is not properly
handled
when you do not execute the package goal.
Stephan
Am 25.02.2015 16:54 schrieb Max Michels m...@apache.org:
Hi Matthias,
Did you really pull from the latest master? I just tried to compile
flink-streaming-examples using mvn clean compile test-compile and it
worked.
Best regards,
Max
On Wed, Feb 25, 2015 at 4:13 PM, Matthias J. Sax
mj...@informatik.hu-berlin.de wrote:
Hi,
I tried to build flink-streaming-examples, but got an compilation
error.
If I build flink-streaming (that implicitly builds
flink-streaming-examples), it works.
I tried it on a fresh clone of flink/master using mvn clean compile
test-compile
Can anybody explain this behavior?
-Matthias
mjsax@T420s-dbis-mjsax
:~/workspace_flink/dummy/flink/flink-staging/flink-streaming/flink-streaming-examples$
mvn clean compile test-compile
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the
effective
model for org.apache.flink:flink-streaming-examples:jar:0.9-SNAPSHOT
[WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique
but found duplicate declaration of plugin
org.apache.maven.plugins:maven-jar-plugin @ line 462, column 12
[WARNING]
[WARNING] It is highly recommended to fix these problems because
they
threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer
support building such malformed projects.
[WARNING]
[INFO]
[INFO]
[INFO] Building flink-streaming-examples 0.9-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @
flink-streaming-examples ---
[INFO] Deleting
/home/mjsax/workspace_flink/dummy/flink/flink-staging/flink-streaming/flink-streaming-examples/target
[INFO]
[INFO] --- maven-checkstyle-plugin:2.12.1:check (validate) @
flink-streaming-examples ---
[INFO]
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-maven) @
flink-streaming-examples ---
[INFO]
[INFO] --- build-helper-maven-plugin:1.7:add-source (add-source) @
flink-streaming-examples ---
[INFO] Source directory:
/home/mjsax/workspace_flink/dummy/flink/flink-staging/flink-streaming/flink-streaming-examples/src/main/scala
added.
[INFO]
[INFO] ---
Re: Contributing to Flink
Hi Niraj, Thanks for your interest at Apache Flink. The quickest is to just give Flink a spin and figure out how it works. This would get you start on how it works before actually doing work on Flink =) Please do visit Flink how to contribute page [1] and subscribe to dev mailing list [2] to start following up. Welcome =) [1] http://flink.apache.org/how-to-contribute.html [2] http://flink.apache.org/community.html#mailing-lists On Thu, Feb 26, 2015 at 1:45 PM, Rai, Niraj niraj@intel.com wrote: Hi Flink Dev, I am looking to contribute to Flink, especially in the area of security. In the past, I have contributed to Pig, Hive and HDFS. I would really appreciate, if I can get some work assigned to me. Looking forward to hear back from the development community of Flink. Thanks Niraj
Re: Drop support for CDH4 / Hadoop 2.0.0-alpha
If we were to drop CDH4 / Hadoop 2.0.0-alpha, would this mean we do not even to shade the hadoop fat jars, or we do still needed to support 1.x ? - Henry On Thu, Feb 26, 2015 at 8:57 AM, Robert Metzger rmetz...@apache.org wrote: Hi, I'm currently working on https://issues.apache.org/jira/browse/FLINK-1605 and its a hell of a mess. I got almost everything working, except for the hadoop 2.0.0-alpha profile. The profile exists because google protobuf has a different version in that Hadoop release. Since maven is setting the version of protobuf for the entire project to the older version, we have to use an older akka version which is causing issues. The logical conclusion from that would be shading Hadoop's protobuf version into the Hadoop jars. That by itself is working, however its not working for the flink-yarn-tests. I think I can also solve the issue with the flink-yarn-tests, but it would be a very dirty hack (either injecting shaded code into the failsafe tests-classpath or putting test code into src/main). But the general question remains: Are we willing to continue spending a lot of time on maintaining the profile? Till has spend a lot of time recently to fix failing testcases for that old akka version, I spend almost two days now on getting the shading/dependencies right, and I'm sure we'll keep having troubles with the profile. Therefore, I was wondering if this is the right time to drop support for CDH4 / Hadoop 2.0.0-alpha. Best, Robert
