Cool. Thanks for the update. I think you can host the ZeroMQ connectors on a private repository or so.
"by the way Spark has LGPL licensed packages in its NOTICE" --> did you find the discussion in their mailing list / JIRA regarding this? Maybe they contacted the authors of the code or got a special permission to do that? What is the license of this file? https://github.com/mbalassi/incubator-flink/blob/streaming-ready/flink-addons/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/index/BTreeIndex.java The build errors for hadoop1 are: [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project flink-streaming-core: Compilation failure: Compilation failure: [ERROR] /home/travis/build/mbalassi/incubator-flink/flink-addons/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/streamcomponent/AbstractStreamComponent.java:[190,89] incompatible types: java.lang.Class<org.apache.flink.streaming.api.streamrecord.StreamRecord> cannot be converted to java.lang.Class<? extends org.apache.flink.streaming.api.streamrecord.StreamRecord<IN>> [ERROR] /home/travis/build/mbalassi/incubator-flink/flink-addons/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/streamcomponent/AbstractStreamComponent.java:[208,42] incompatible types: java.lang.Class<org.apache.flink.streaming.partitioner.DefaultPartitioner> cannot be converted to java.lang.Class<? extends org.apache.flink.runtime.io.network.api.ChannelSelector<org.apache.flink.streaming.api.streamrecord.StreamRecord<OUT>>> I guess thats easy to fix ? The build errors for hadoop2 are: I think its copypasta.. You've copied the build-profiles stuff from another project. Your pom is including the "flink-hbase" and "flink-yarn" submodules (of flink-streaming). https://github.com/mbalassi/incubator-flink/blob/streaming-ready/flink-addons/flink-streaming/pom.xml#L109 Just remove the whole <profiles> .. </profiles> block in the flink-streaming/pom.xml. On Wed, Jul 16, 2014 at 1:56 PM, Márton Balassi <[email protected]> wrote: > Hi all, > > We've decided to do our preparations on a fork of the main repo: > *https://github.com/mbalassi/incubator-flink/tree/streaming-ready > <https://github.com/mbalassi/incubator-flink/tree/streaming-ready>* > > We've fixed the code to match the coding style and added the modules to the > maven build. > > https://github.com/mbalassi/incubator-flink/commit/f8a6b0ecf7f453cad13ed6752051f29783ec0469 > > As for the licensing: > > https://github.com/mbalassi/incubator-flink/commit/5ddcebc6f0cedfcb3ed67a4f53ee1b415dd1d82f > > - Removed JBlas as it is no longer needed > - Included the information for RabbitMQ > - Deleted the ZeroMQ package and it's dependency as a whole - by the way > Spark has LGPL licensed packages in its NOTICE > - Did not include additional information for Apache Kafka > > On my machine the project builds with the default hadoop profile and Java > 6&7 and the tests are passing, however the Travis CI for the latest travis > build is way less rosy: > *https://travis-ci.org/mbalassi/incubator-flink/builds/30055789 > <https://travis-ci.org/mbalassi/incubator-flink/builds/30055789>* > > - The ones with the hadoop-2 profile fail with not finding one of the > poms (?) > - The ones with the hadoop-1 profile either fail in flink-tests with an > error in DataSink (maybe an the travis slot run out of disk...) or an > exception in the streaming code that did not occur when neither when I > built the project locally with maven nor in Eclipse > > Do you have any suggestion for additional requirements or fixing the CI > build? > > Cheers, > > Marton > > > On Mon, Jul 14, 2014 at 6:28 PM, Márton Balassi <[email protected]> > wrote: > > > Hi guys, > > > > @Stefan: Thanks for the script, we've gone through the commits with > Gabor, > > Gyula is reviewing it right now. > > https://github.com/mbalassi/incubator-flink/commits/streamrebase3 > > > > @Robert: We've went through the coding style, the update commit is > already > > pushed to our old repo, I'm merging it to my flink fork soon. > > > > @Henry: Ok, I'm pinging all the contributors with the subject, the three > > of us already signed the form. > > > > I'm dealing with the Licensing tomorrow. > > > > > > On Mon, Jul 14, 2014 at 4:58 PM, Stephan Ewen <[email protected]> wrote: > > > >> Before adding this contribution to the project, there are some legal > >> things > >> to do: > >> > >> - Obtain ICLAs from all major contributors. There are 7 in the > streaming > >> code, out of which three did the largest portion of the work: Márton > >> Balassi, Gyula Fóra, Hermann Gábor > >> - @mentors: Should the other 4 also sign and send ICLAs? > >> > >> - Licenses: Walk through the code, collect all dependencies and make > sure > >> they are ASL compatible.Here are some links with information: > >> - http://www.apache.org/legal/resolved.html > >> - http://www.apache.org/foundation/license-faq.html#WhatDoesItMEAN > >> > >> - All used licenses must be mentioned in the LICENSE files > >> - under ./LICENSE > >> - under ./flink-dist/src/main/flink-bin/LICENSE > >> > >> - Check headers for ASF compliance. > >> > >> > >> This looks manageable. Anything I forgot? > >> > >> Greetings, > >> Stephan > >> > >> > >> > >> > >> On Mon, Jul 14, 2014 at 4:43 PM, Stephan Ewen <[email protected]> wrote: > >> > >> > Ho guys! > >> > > >> > I made a scripted manual rebase of each commit (basically add the > commit > >> > not via its diff, but such that it reflects the code base after the > >> commit) > >> > > >> > https://github.com/StephanEwen/incubator-flink/commits/streamrebase > >> > > >> > No more merge commits that mess things up. You should be able to > squash > >> > things easily via "git rebase -i > >> 3002258f8a22a8adbdb230e57c972ad17910debf" > >> > > >> > The commit diffs may be a bit different than before (not too much if I > >> did > >> > things correctly), but can you have a quick look at the commits to see > >> > whether they make sense? > >> > > >> > Stephan > >> > > >> > > >> > BTW: I used this way to do it: > >> > > >> > Have two repositories (clones) > >> > - /data/repositories/flink > >> > - /data/repositories/flinkbak > >> > > >> > The do the following for every non-merge commit: > >> > - Check out the state after a commit in the backup (detached head) > >> > - Remove current streaming directory (physically and from the index) > >> > - Add it again (files and index), with the state of the cloned repo > >> > - Commit (git recreates the diffs in a way that they reflect the > >> original > >> > commit plus any merges) > >> > > >> > --------------------- > >> > > >> > #!/bin/bash > >> > > >> > for line in $(cat commits) > >> > do > >> > cd /data/repositories/flinkbak > >> > author=`git --no-pager show -s --format='%an <%ae>' $line` > >> > message=`git --no-pager show -s --format='%s%n' $line` > >> > > >> > echo "picking commit $line from author $author" > >> > > >> > git checkout $line > >> > cd /data/repositories/flink > >> > rm -rf "/data/repositories/flink/flink-addons/flink-streaming" > >> > git rm -r "/data/repositories/flink/flink-addons/flink-streaming" > >> > cp -r "/data/repositories/flinkbak/flink-addons/flink-streaming" > >> > "/data/repositories/flink/flink-addons/flink-streaming" > >> > git add /data/repositories/flink/flink-addons/flink-streaming > >> > git commit --author "$author" --m "$message" > >> > > >> > # read -rsp $'Press any key to continue...\n' -n1 key > >> > done > >> > > >> > > >> > > >> > > >> > > >> > On Mon, Jul 14, 2014 at 1:10 PM, Gyula Fóra <[email protected]> > >> wrote: > >> > > >> >> By the way, I forked your repo switch to the streaming branch and > then > >> I > >> >> executed the commands (I think this is how it should have been done) > >> >> > >> >> > >> >> On Mon, Jul 14, 2014 at 1:09 PM, Gyula Fóra <[email protected]> > >> wrote: > >> >> > >> >>> This is what I get with "rebase -i -p master": > >> >>> > >> >>> pick 9456624 Merge branch 'master' of > >> file:///data/repositories/streamin > >> >>> into streaming > >> >>> pick 89299b8 [streaming] Post-merge cleanups > >> >>> > >> >>> #Rebase 1fd457d..89299b8 onto 1fd457d > >> >>> #...... > >> >>> > >> >>> > >> >>> On Mon, Jul 14, 2014 at 12:47 PM, Stephan Ewen <[email protected]> > >> wrote: > >> >>> > >> >>>> Can you do "rebase -i -p master". That should include all commits > and > >> >>>> might save you the meeting hell. > >> >>>> > >> >>> > >> >>> > >> >> > >> > > >> > > > > >
