+1. And thanks a lot to Chesnay for pushing this.
Best, Hequn On Thu, Jul 4, 2019 at 8:07 PM Chesnay Schepler <ches...@apache.org> wrote: > Note that the Flinkbot approach isn't that trivial either; we can't > _just_ trigger builds for a branch in the apache repo, but would first > have to clone the branch/pr into a separate repository (that is owned by > the github account that the travis account would be tied to). > > One roadblock after the next showing up... > > On 04/07/2019 11:59, Chesnay Schepler wrote: > > Small update with mostly bad news: > > > > INFRA doesn't know whether it is possible, and referred my to Travis > > support. > > They did point out that it could be problematic in regards to > > read/write permissions for the repository. > > > > From my own findings /so far/ with a test repo/organization, it does > > not appear possible to configure the Travis account used for a > > specific repository. > > > > So yeah, if we go down this route we may have to pimp the Flinkbot to > > trigger builds through the Travis REST API. > > > > On 04/07/2019 10:46, Chesnay Schepler wrote: > >> I've raised a JIRA > >> <https://issues.apache.org/jira/browse/INFRA-18703>with INFRA to > >> inquire whether it would be possible to switch to a different Travis > >> account, and if so what steps would need to be taken. > >> We need a proper confirmation from INFRA since we are not in full > >> control of the flink repository (for example, we cannot access the > >> settings page). > >> > >> If this is indeed possible, Ververica is willing sponsor a Travis > >> account for the Flink project. > >> This would provide us with more than enough resources than we need. > >> > >> Since this makes the project more reliant on resources provided by > >> external companies I would like to vote on this. > >> > >> Please vote on this proposal, as follows: > >> [ ] +1, Approve the migration to a Ververica-sponsored Travis > >> account, provided that INFRA approves > >> [ ] -1, Do not approach the migration to a Ververica-sponsored Travis > >> account > >> > >> The vote will be open for at least 24h, and until we have > >> confirmation from INFRA. The voting period may be shorter than the > >> usual 3 days since our current is effectively not working. > >> > >> On 04/07/2019 06:51, Bowen Li wrote: > >>> Re: > Are they using their own Travis CI pool, or did the switch to > >>> an entirely different CI service? > >>> > >>> I reached out to Wes and Krisztián from Apache Arrow PMC. They are > >>> currently moving away from ASF's Travis to their own in-house metal > >>> machines at [1] with custom CI application at [2]. They've seen > >>> significant improvement w.r.t both much higher performance and > >>> basically no resource waiting time, "night-and-day" difference > >>> quoting Wes. > >>> > >>> Re: > If we can just switch to our own Travis pool, just for our > >>> project, then this might be something we can do fairly quickly? > >>> > >>> I believe so, according to [3] and [4] > >>> > >>> > >>> [1] https://ci.ursalabs.org/ <https://ci.ursalabs.org/#/> > >>> [2] https://github.com/ursa-labs/ursabot > >>> [3] > >>> > https://docs.travis-ci.com/user/migrate/open-source-repository-migration > >>> > >>> [4] > >>> https://docs.travis-ci.com/user/migrate/open-source-on-travis-ci-com > >>> > >>> > >>> > >>> On Wed, Jul 3, 2019 at 12:01 AM Chesnay Schepler <ches...@apache.org > >>> <mailto:ches...@apache.org>> wrote: > >>> > >>> Are they using their own Travis CI pool, or did the switch to an > >>> entirely different CI service? > >>> > >>> If we can just switch to our own Travis pool, just for our > >>> project, then > >>> this might be something we can do fairly quickly? > >>> > >>> On 03/07/2019 05:55, Bowen Li wrote: > >>> > I responded in the INFRA ticket [1] that I believe they are > >>> using a wrong > >>> > metric against Flink and the total build time is a completely > >>> different > >>> > thing than guaranteed build capacity. > >>> > > >>> > My response: > >>> > > >>> > "As mentioned above, since I started to pay attention to Flink's > >>> build > >>> > queue a few tens of days ago, I'm in Seattle and I saw no build > >>> was kicking > >>> > off in PST daytime in weekdays for Flink. Our teammates in China > >>> and Europe > >>> > have also reported similar observations. So we need to evaluate > >>> how the > >>> > large total build time came from - if 1) your number and 2) our > >>> > observations from three locations that cover pretty much a full > >>> day, are > >>> > all true, I **guess** one reason can be that - highly likely the > >>> extra > >>> > build time came from weekends when other Apache projects may be > >>> idle and > >>> > Flink just drains hard its congested queue. > >>> > > >>> > Please be aware of that we're not complaining about the lack of > >>> resources > >>> > in general, I'm complaining about the lack of **stable, > >>> dedicated** > >>> > resources. An example for the latter one is, currently even if > >>> no build is > >>> > in Flink's queue and I submit a request to be the queue head > >>> in PST > >>> > morning, my build won't even start in 6-8+h. That is an absurd > >>> amount of > >>> > waiting time. > >>> > > >>> > That's saying, if ASF INFRA decides to adopt a quota system and > >>> grants > >>> > Flink five DEDICATED servers that runs all the time only for > >>> Flink, that'll > >>> > be PERFECT and can totally solve our problem now. > >>> > > >>> > Please be aware of that we're not complaining about the lack of > >>> resources > >>> > in general, I'm complaining about the lack of **stable, > >>> dedicated** > >>> > resources. An example for the latter one is, currently even if > >>> no build is > >>> > in Flink's queue and I submit a request to be the queue head > >>> in PST > >>> > morning, my build won't even start in 6-8+h. That is an absurd > >>> amount of > >>> > waiting time. > >>> > > >>> > > >>> > That's saying, if ASF INFRA decides to adopt a quota system and > >>> grants > >>> > Flink five DEDICATED servers that runs all the time only for > >>> Flink, that'll > >>> > be PERFECT and can totally solve our problem now. > >>> > > >>> > I feel what's missing in the ASF INFRA's Travis resource pool is > >>> some level > >>> > of build capacity SLAs and certainty" > >>> > > >>> > > >>> > Again, I believe there are differences in nature of these two > >>> problems, > >>> > long build time v.s. lack of dedicated build resource. That's > >>> saying, > >>> > shortening build time may relieve the situation, and may not. > >>> I'm sightly > >>> > negative on disabling IT cases for PRs, due to the downside is > >>> that we are > >>> > at risk of any potential bugs in PR that UTs doesn't catch, and > >>> may cost a > >>> > lot more to fix and if it slows others down or even block > >>> others, but am > >>> > open to others opinions on it. > >>> > > >>> > AFAICT from INFRA ticket[1], donating to ASF INFRA won't be > >>> feasible to > >>> > solve our problem since INFRA's pool is fully shared and they > >>> have no > >>> > control and finer insights over resource allocation to a > >>> specific Apache > >>> > project. As mentioned in [1], Apache Arrow is moving away from > >>> ASF INFRA > >>> > Travis pool (they are actually surprised Flink hasn't plan to do > >>> so). I > >>> > know that Spark is on its own build infra. If we all agree that > >>> funding our > >>> > own build infra, I'd be glad to help investigate any potential > >>> options > >>> > after releasing 1.9 since I'm super busy with 1.9 now. > >>> > > >>> > [1] https://issues.apache.org/jira/browse/INFRA-18533 > >>> > > >>> > > >>> > > >>> > On Tue, Jul 2, 2019 at 4:46 AM Chesnay Schepler > >>> <ches...@apache.org <mailto:ches...@apache.org>> wrote: > >>> > > >>> >> As a short-term stopgap, since we can assume this issue to > >>> become much > >>> >> worse in the following days/weeks, we could disable IT cases in > >>> PRs and > >>> >> only run them on master. > >>> >> > >>> >> On 02/07/2019 12:03, Chesnay Schepler wrote: > >>> >>> People really have to stop thinking that just because > >>> something works > >>> >>> for us it is also a good solution. > >>> >>> Also, please remember that our builds run for 2h from start to > >>> finish, > >>> >>> and not the 14 _minutes_ it takes for zeppelin. > >>> >>> We are dealing with an entirely different scale here, both in > >>> terms of > >>> >>> build times and number of builds. > >>> >>> > >>> >>> In this very thread people have been complaining about long > >>> queue > >>> >>> times for their builds. Surprise, other Apache projects have > >>> been > >>> >>> suffering the very same thing due to us not controlling our > >>> build > >>> >>> times. While switching services (be it Jenkins, CircleCI or > >>> whatever) > >>> >>> will possibly work for us (and these options are actually > >>> attractive, > >>> >>> like CircleCI's proper support for build artifacts), it will > >>> also > >>> >>> result in us likely negatively affecting other projects in > >>> significant > >>> >>> ways. > >>> >>> > >>> >>> Sure, the Jenkins setup has a good user experience for us, at > >>> the cost > >>> >>> of blocking Jenkins workers for a _lot_ of time. Right now we > >>> have 25 > >>> >>> PR's in our queue; that's possibly 50h we'd consume of Jenkins > >>> >>> resources, and the European contributors haven't even really > >>> started yet. > >>> >>> > >>> >>> FYI, the latest INFRA response from INFRA-18533: > >>> >>> > >>> >>> "Our rough metrics shows that Flink used over 5800 hours of > >>> build time > >>> >>> last month. That is equal to EIGHT servers running 24/7 for > >>> the ENTIRE > >>> >>> MONTH. EIGHT. nonstop. > >>> >>> When we discovered this last night, we discussed it some and > >>> are going > >>> >>> to tune down Flink to allow only five executors maximum. We > >>> cannot > >>> >>> allow Flink to consume so much of a Foundation shared > >>> resource." > >>> >>> > >>> >>> So yes, we either > >>> >>> a) have to heavily reduce our CI usage or > >>> >>> b) fund our own, either maintaining it ourselves or donating > >>> to Apache. > >>> >>> > >>> >>> On 02/07/2019 05:11, Bowen Li wrote: > >>> >>>> By looking at the git history of the Jenkins script, its core > >>> part > >>> >>>> was finished in March 2017 (and only two minor update in > >>> 2017/2018), > >>> >>>> so it's been running for over two years now and feels like > >>> Zepplin > >>> >>>> community has been quite happy with it. @Jeff Zhang > >>> >>>> <mailto:zjf...@gmail.com <mailto:zjf...@gmail.com>> can you > >>> share your insights and user > >>> >>>> experience with the Jenkins+Travis approach? > >>> >>>> > >>> >>>> Things like: > >>> >>>> > >>> >>>> - has the approach completely solved the resource capacity > >>> problem > >>> >>>> for Zepplin community? is Zepplin community happy with the > >>> result? > >>> >>>> - is the whole configuration chain stable (e.g. uptime) > >>> enough? > >>> >>>> - how often do you need to maintain the Jenkins infra? how > >>> many > >>> >>>> people are usually involved in maintenance and bug-fixes? > >>> >>>> > >>> >>>> The downside of this approach seems mostly to be on the > >>> maintenance > >>> >>>> to me - maintain the script and Jenkins infra. > >>> >>>> > >>> >>>> ** Having Our Own Travis-CI.com Account ** > >>> >>>> > >>> >>>> Another alternative I've been thinking of is to have our own > >>> >>>> travis-ci.com <http://travis-ci.com> <http://travis-ci.com> > >>> account with paid dedicated > >>> >>>> resources. Note travis-ci.org <http://travis-ci.org> > >>> <http://travis-ci.org> is the free > >>> >>>> version and travis-ci.com <http://travis-ci.com> > >>> <http://travis-ci.com> is the commercial > >>> >>>> version. We currently use a shared resource pool managed by > >>> ASK INFRA > >>> >>>> team on travis-ci.org <http://travis-ci.org> > >>> <http://travis-ci.org>, but we have no control > >>> >>>> over it - we can't see how it's configured, how much > >>> resources are > >>> >>>> available, how resources are allocated among Apache projects, > >>> etc. > >>> >>>> The nice thing about having an account on travis-ci.com > >>> <http://travis-ci.com> > >>> >>>> <http://travis-ci.com> are: > >>> >>>> > >>> >>>> - relatively low cost with much better resource guarantee > >>> than what > >>> >>>> we currently have [1]: $249/month with 5 dedicated > >>> concurrency, > >>> >>>> $489/month with 10 concurrency > >>> >>>> - low maintenance work compared to using Jenkins > >>> >>>> - (potentially) no migration cost according to Travis's doc > >>> [2] > >>> >>>> (pending verification) > >>> >>>> - full control over the build capacity/configuration > >>> compared to > >>> >>>> using ASF INFRA's pool > >>> >>>> > >>> >>>> I'd be surprised if we as such a vibrant community cannot > >>> find and > >>> >>>> fund $249*12=$2988 a year in exchange for a much better > >>> developer > >>> >>>> experience and much higher productivity. > >>> >>>> > >>> >>>> [1] https://travis-ci.com/plans > >>> >>>> [2] > >>> >>>> > >>> >> > >>> > https://docs.travis-ci.com/user/migrate/open-source-repository-migration > >>> > >>> >>>> On Sat, Jun 29, 2019 at 8:39 AM Chesnay Schepler > >>> <ches...@apache.org <mailto:ches...@apache.org> > >>> >>>> <mailto:ches...@apache.org <mailto:ches...@apache.org>>> > >>> wrote: > >>> >>>> > >>> >>>> So yes, the Jenkins job keeps pulling the state from > >>> Travis until it > >>> >>>> finishes. > >>> >>>> > >>> >>>> Note sure I'm comfortable with the idea of using Jenkins > >>> workers > >>> >>>> just to > >>> >>>> idle for a several hours. > >>> >>>> > >>> >>>> On 29/06/2019 14:56, Jeff Zhang wrote: > >>> >>>> > Here's what zeppelin community did, we make a python > >>> script to > >>> >>>> check the > >>> >>>> > build status of pull request. > >>> >>>> > Here's script: > >>> >>>> > > >>> https://github.com/apache/zeppelin/blob/master/travis_check.py > >>> >>>> > > >>> >>>> > And this is the script we used in Jenkins build job. > >>> >>>> > > >>> >>>> > if [ -f "travis_check.py" ]; then > >>> >>>> > git log -n 1 > >>> >>>> > STATUS=$(curl -s $BUILD_URL | grep -e "GitHub pull > >>> >>>> request.*from.*" | sed > >>> >>>> > 's/.*GitHub pull request <a > >>> >>>> > href=\"\(https[^"]*\).*from[^"]*.\(https[^"]*\).*/\1 > >>> \2/g') > >>> >>>> > AUTHOR=$(echo $STATUS | sed 's/.*[/]\(.*\)$/\1/g') > >>> >>>> > PR=$(echo $STATUS | awk '{print $1}' | sed > >>> >>>> 's/.*[/]\(.*\)$/\1/g') > >>> >>>> > #COMMIT=$(git log -n 1 | grep "^Merge:" | awk > >>> '{print $3}') > >>> >>>> > #if [ -z $COMMIT ]; then > >>> >>>> > # COMMIT=$(curl -s > >>> >>>> https://api.github.com/repos/apache/zeppelin/pulls/$PR > >>> >>>> > | grep -e "\"label\":" -e "\"ref\":" -e "\"sha\":" | > >>> tr '\n' ' ' > >>> >>>> | sed > >>> >>>> > 's/\(.*sha[^,]*,\)\(.*ref.*\)/\1 = \2/g' | tr = '\n' | > >>> grep -v > >>> >>>> "apache:" | > >>> >>>> > sed 's/.*sha.[^"]*["]\([^"]*\).*/\1/g') > >>> >>>> > #fi > >>> >>>> > > >>> >>>> > # get commit hash from PR > >>> >>>> > COMMIT=$(curl -s > >>> >>>> https://api.github.com/repos/apache/zeppelin/pulls/$PR | > >>> >>>> > grep -e "\"label\":" -e "\"ref\":" -e "\"sha\":" | tr > >>> '\n' ' ' > >>> >>>> | sed > >>> >>>> > 's/\(.*sha[^,]*,\)\(.*ref.*\)/\1 = \2/g' | tr = '\n' | > >>> grep -v > >>> >>>> "apache:" | > >>> >>>> > sed 's/.*sha.[^"]*["]\([^"]*\).*/\1/g') > >>> >>>> > sleep 30 # sleep few moment to wait travis starts > >>> the build > >>> >>>> > RET_CODE=0 > >>> >>>> > python ./travis_check.py ${AUTHOR} ${COMMIT} || > >>> RET_CODE=$? > >>> >>>> > if [ $RET_CODE -eq 2 ]; then # try with repository > >>> name when > >>> >>>> travis-ci is > >>> >>>> > not available in the account > >>> >>>> > RET_CODE=0 > >>> >>>> > AUTHOR=$(curl -s > >>> >>>> https://api.github.com/repos/apache/zeppelin/pulls/$PR > >>> >>>> > | grep '"full_name":' | grep -v "apache/zeppelin" | sed > >>> >>>> > 's/.*[:][^"]*["]\([^/]*\).*/\1/g') > >>> >>>> > python ./travis_check.py ${AUTHOR} ${COMMIT} || > >>> RET_CODE=$? > >>> >>>> > fi > >>> >>>> > > >>> >>>> > if [ $RET_CODE -eq 2 ]; then # fail with can't find > >>> build > >>> >>>> information in > >>> >>>> > the travis > >>> >>>> > set +x > >>> >>>> > echo > >>> "-----------------------------------------------------" > >>> >>>> > echo "Looks like travis-ci is not configured for > >>> your fork." > >>> >>>> > echo "Please setup by swich on 'zeppelin' > >>> repository at > >>> >>>> > https://travis-ci.org/profile and travis-ci." > >>> >>>> > echo "And then make sure 'Build branch updates' > >>> option is > >>> >>>> enabled in > >>> >>>> > the settings > >>> https://travis-ci.org/${AUTHOR}/zeppelin/settings > >>> <https://travis-ci.org/$%7BAUTHOR%7D/zeppelin/settings> > >>> >>>> <https://travis-ci.org/$%7BAUTHOR%7D/zeppelin/settings>." > >>> >>>> > echo "" > >>> >>>> > echo "To trigger CI after setup, you will need > >>> ammend your > >>> >>>> last commit > >>> >>>> > with" > >>> >>>> > echo "git commit --amend" > >>> >>>> > echo "git push your-remote HEAD --force" > >>> >>>> > echo "" > >>> >>>> > echo "See > >>> >>>> > > >>> >>>> > >>> >> > >>> > http://zeppelin.apache.org/contribution/contributions.html#continuous-integration > >>> >>>> > ." > >>> >>>> > fi > >>> >>>> > > >>> >>>> > exit $RET_CODE > >>> >>>> > else > >>> >>>> > set +x > >>> >>>> > echo "travis_check.py does not exists" > >>> >>>> > exit 1 > >>> >>>> > fi > >>> >>>> > > >>> >>>> > Chesnay Schepler <ches...@apache.org > >>> <mailto:ches...@apache.org> > >>> >>>> <mailto:ches...@apache.org <mailto:ches...@apache.org>>> > >>> 于2019年6月29日周六 下午3:17写道： > >>> >>>> > > >>> >>>> >> Does this imply that a Jenkins job is active as long > >>> as the > >>> >>>> Travis build > >>> >>>> >> runs? > >>> >>>> >> > >>> >>>> >> On 26/06/2019 21:28, Bowen Li wrote: > >>> >>>> >>> Hi, > >>> >>>> >>> > >>> >>>> >>> @Dawid, I think the "long test running" as I > >>> mentioned in the > >>> >>>> first > >>> >>>> >> email, > >>> >>>> >>> also as you guys said, belongs to "a big effort > >>> which is much > >>> >>>> harder to > >>> >>>> >>> accomplish in a short period of time and may deserve > >>> its own > >>> >>>> separate > >>> >>>> >>> discussion". Thus I didn't include it in what we can > >>> do in a > >>> >>>> foreseeable > >>> >>>> >>> short term. > >>> >>>> >>> > >>> >>>> >>> Besides, I don't think that's the ultimate reason > >>> for lack of > >>> >>>> build > >>> >>>> >>> resources. Even if the build is shortened to > >>> something like > >>> >>>> 2h, the > >>> >>>> >>> problems of no build machine works about 6 or more > >>> hours in > >>> >>>> PST daytime > >>> >>>> >>> that I described will still happen, because no > >>> machine from > >>> >>>> ASF INFRA's > >>> >>>> >>> pool is allocated to Flink. As I have paid close > >>> attention to > >>> >>>> the build > >>> >>>> >>> queue in the past few weekdays, it's a pretty clear > >>> pattern now. > >>> >>>> >>> > >>> >>>> >>> **The ultimate root cause** for that is - we don't > >>> have any > >>> >>>> **dedicated** > >>> >>>> >>> build resources that we can stably rely on. I'm > >>> actually ok to > >>> >>>> wait for a > >>> >>>> >>> long time if there are build requests running, it > >>> means at > >>> >>>> least we are > >>> >>>> >>> making progress. But I'm not ok with no build > >>> resource. A > >>> >>>> better place I > >>> >>>> >>> think we should aim at in short term is to always > >>> have at > >>> >>>> least a central > >>> >>>> >>> pool (can be 3 or 5) of machines dedicated to build > >>> Flink at > >>> >>>> any time, or > >>> >>>> >>> maybe use users resources. > >>> >>>> >>> > >>> >>>> >>> @Chesnay @Robert I synced with Jeff offline that > >>> Zeppelin > >>> >>>> community is > >>> >>>> >>> using a Jenkins job to automatically build on users' > >>> travis > >>> >>>> account and > >>> >>>> >>> link the result back to github PR. I guess the > >>> Jenkins job > >>> >>>> would fetch > >>> >>>> >>> latest upstream master and build the PR against it. > >>> Jeff has > >>> >>>> filed > >>> >>>> >> tickets > >>> >>>> >>> to learn and get access to the Jenkins infra. It'll > >>> better to > >>> >>>> fully > >>> >>>> >>> understand it first before judging this approach. > >>> >>>> >>> > >>> >>>> >>> I also heard good things about CircleCI, and ASF > >>> INFRA seems > >>> >>>> to have a > >>> >>>> >> pool > >>> >>>> >>> of build capacity there too. Can be an alternative > >>> to consider. > >>> >>>> >>> > >>> >>>> >>> > >>> >>>> >>> > >>> >>>> >>> > >>> >>>> >>> > >>> >>>> >>> > >>> >>>> >>> > >>> >>>> >>> > >>> >>>> >>> > >>> >>>> >>> On Wed, Jun 26, 2019 at 12:44 AM Dawid Wysakowicz < > >>> >>>> >> dwysakow...@apache.org > >>> <mailto:dwysakow...@apache.org> <mailto:dwysakow...@apache.org > >>> <mailto:dwysakow...@apache.org>>> > >>> >>>> >>> wrote: > >>> >>>> >>> > >>> >>>> >>>> Sorry to jump in late, but I think Bowen missed the > >>> most > >>> >>>> important point > >>> >>>> >>>> from Chesnay's previous message in the summary. The > >>> ultimate > >>> >>>> reason for > >>> >>>> >>>> all the problems is that the tests take close to 2 > >>> hours to > >>> >>>> run already. > >>> >>>> >>>> I fully support this claim: "Unless people start > >>> caring about > >>> >>>> test times > >>> >>>> >>>> before adding them, this issue cannot be solved" > >>> >>>> >>>> > >>> >>>> >>>> This is also another reason why using user's Travis > >>> account > >>> >>>> won't help. > >>> >>>> >>>> Every few weeks we reach the user's time limit for > >>> a single > >>> >>>> profile. > >>> >>>> >>>> This makes the user's builds simply fail, until we > >>> either > >>> >>>> properly > >>> >>>> >>>> decrease the time the tests take (which I am not > >>> sure we ever > >>> >>>> did) or > >>> >>>> >>>> postpone the problem by splitting into more > >>> profiles. (Note > >>> >>>> that the ASF > >>> >>>> >>>> Travis account has higher time limits) > >>> >>>> >>>> > >>> >>>> >>>> Best, > >>> >>>> >>>> > >>> >>>> >>>> Dawid > >>> >>>> >>>> > >>> >>>> >>>> On 26/06/2019 09:36, Robert Metzger wrote: > >>> >>>> >>>>> Do we know if using "the best" available hardware > >>> would > >>> >>>> improve the > >>> >>>> >> build > >>> >>>> >>>>> times? > >>> >>>> >>>>> Imagine we would run the build on machines with > >>> plenty of > >>> >>>> main memory > >>> >>>> >> to > >>> >>>> >>>>> mount everything to ramdisk + the latest CPU > >>> architecture? > >>> >>>> >>>>> > >>> >>>> >>>>> Throwing hardware at the problem could help reduce > >>> the time > >>> >>>> of an > >>> >>>> >>>>> individual build, and using our own infrastructure > >>> would > >>> >>>> remove our > >>> >>>> >>>>> dependency on Apache's Travis account (with the > >>> obvious > >>> >>>> downside of > >>> >>>> >>>> having > >>> >>>> >>>>> to maintain the infrastructure) > >>> >>>> >>>>> We could use an open source travis alternative, to > >>> have a > >>> >>>> similar > >>> >>>> >>>>> experience and make the migration easy. > >>> >>>> >>>>> > >>> >>>> >>>>> > >>> >>>> >>>>> On Wed, Jun 26, 2019 at 9:34 AM Chesnay Schepler > >>> >>>> <ches...@apache.org <mailto:ches...@apache.org> > >>> <mailto:ches...@apache.org <mailto:ches...@apache.org>>> > >>> >>>> >>>> wrote: > >>> >>>> >>>>>> >From what I gathered, there's no special > >>> sauce that the > >>> >>>> Zeppelin > >>> >>>> >>>>>> project uses which actually integrates a users > >>> Travis > >>> >>>> account into the > >>> >>>> >>>> PR. > >>> >>>> >>>>>> They just disabled Travis for PRs. And that's > >>> kind of it. > >>> >>>> >>>>>> > >>> >>>> >>>>>> Naturally we can do this (duh) and safe the ASF a > >>> fair > >>> >>>> amount of > >>> >>>> >>>>>> resources, but there are downsides: > >>> >>>> >>>>>> > >>> >>>> >>>>>> The discoverability of the Travis check takes a > >>> nose-dive. > >>> >>>> Either we > >>> >>>> >>>>>> require every contributor to always, an every > >>> commit, also > >>> >>>> post a > >>> >>>> >> Travis > >>> >>>> >>>>>> build, or we have the reviewer sift through the > >>> >>>> contributors account > >>> >>>> >> to > >>> >>>> >>>>>> find it. > >>> >>>> >>>>>> > >>> >>>> >>>>>> This is rather cumbersome. Additionally, it's > >>> also not > >>> >>>> equivalent to > >>> >>>> >>>>>> having a PR build. > >>> >>>> >>>>>> > >>> >>>> >>>>>> A normal branch build takes a branch as is and > >>> tests it. A > >>> >>>> PR build > >>> >>>> >>>>>> merges the branch into master, and then runs it. > >>> (Fun fact: > >>> >>>> This is > >>> >>>> >> why > >>> >>>> >>>>>> a PR without merge conflicts is not being run on > >>> Travis.) > >>> >>>> >>>>>> > >>> >>>> >>>>>> And ultimately, everyone can already make use > >>> of this > >>> >>>> approach anyway. > >>> >>>> >>>>>> > >>> >>>> >>>>>> On 25/06/2019 08:02, Jark Wu wrote: > >>> >>>> >>>>>>> Hi Jeff, > >>> >>>> >>>>>>> > >>> >>>> >>>>>>> Thanks for sharing the Zeppelin approach. I > >>> think it's a > >>> >>>> good idea to > >>> >>>> >>>>>>> leverage user's travis account. > >>> >>>> >>>>>>> In this way, we can have almost unlimited > >>> concurrent build > >>> >>>> jobs and > >>> >>>> >>>>>>> developers can restart build by themselves > >>> (currently only > >>> >>>> committers > >>> >>>> >>>>>>> can restart PR's build). > >>> >>>> >>>>>>> > >>> >>>> >>>>>>> But I'm still not very clear how to integrate > >>> user's > >>> >>>> travis build > >>> >>>> >> into > >>> >>>> >>>>>>> the Flink pull request's build automatically. > >>> Can you > >>> >>>> explain more in > >>> >>>> >>>>>>> detail? > >>> >>>> >>>>>>> > >>> >>>> >>>>>>> Another question: does travis only build > >>> branches for user > >>> >>>> account? > >>> >>>> >>>>>>> My concern is that builds for PRs will rebase > >>> user's > >>> >>>> commits against > >>> >>>> >>>>>>> current master branch. > >>> >>>> >>>>>>> This will help us to find problems before > >>> merge. Builds > >>> >>>> for branches > >>> >>>> >>>>>>> will lose the impact of new commits in master. > >>> >>>> >>>>>>> How does Zeppelin solve this problem? > >>> >>>> >>>>>>> > >>> >>>> >>>>>>> Thanks again for sharing the idea. > >>> >>>> >>>>>>> > >>> >>>> >>>>>>> Regards, > >>> >>>> >>>>>>> Jark > >>> >>>> >>>>>>> > >>> >>>> >>>>>>> On Tue, 25 Jun 2019 at 11:01, Jeff Zhang > >>> <zjf...@gmail.com <mailto:zjf...@gmail.com> > >>> >>>> <mailto:zjf...@gmail.com <mailto:zjf...@gmail.com>> > >>> >>>> >>>>>>> <mailto:zjf...@gmail.com > >>> <mailto:zjf...@gmail.com> <mailto:zjf...@gmail.com > >>> <mailto:zjf...@gmail.com>>>> wrote: > >>> >>>> >>>>>>> > >>> >>>> >>>>>>> Hi Folks, > >>> >>>> >>>>>>> > >>> >>>> >>>>>>> Zeppelin meet this kind of issue before, we > >>> solve > >>> >>>> it by > >>> >>>> >> delegating > >>> >>>> >>>>>>> each > >>> >>>> >>>>>>> one's PR build to his travis account > >>> (Everyone can > >>> >>>> have 5 free > >>> >>>> >>>>>>> slot for > >>> >>>> >>>>>>> travis build). > >>> >>>> >>>>>>> Apache account travis build is only triggered > >>> when > >>> >>>> PR is merged. > >>> >>>> >>>>>>> > >>> >>>> >>>>>>> > >>> >>>> >>>>>>> > >>> >>>> >>>>>>> Kurt Young <ykt...@gmail.com > >>> <mailto:ykt...@gmail.com> > >>> >>>> <mailto:ykt...@gmail.com <mailto:ykt...@gmail.com>> > >>> <mailto:ykt...@gmail.com <mailto:ykt...@gmail.com> > >>> >>>> <mailto:ykt...@gmail.com <mailto:ykt...@gmail.com>>>> > >>> >>>> >>>>>>> 于2019年6月25日周二 上午10:16写道： > >>> >>>> >>>>>>> > >>> >>>> >>>>>>> > (Forgot to cc George) > >>> >>>> >>>>>>> > > >>> >>>> >>>>>>> > Best, > >>> >>>> >>>>>>> > Kurt > >>> >>>> >>>>>>> > > >>> >>>> >>>>>>> > > >>> >>>> >>>>>>> > On Tue, Jun 25, 2019 at 10:16 AM Kurt Young > >>> >>>> <ykt...@gmail.com <mailto:ykt...@gmail.com> > >>> <mailto:ykt...@gmail.com <mailto:ykt...@gmail.com>> > >>> >>>> >>>>>>> <mailto:ykt...@gmail.com > >>> <mailto:ykt...@gmail.com> <mailto:ykt...@gmail.com > >>> <mailto:ykt...@gmail.com>>>> > >>> >>>> wrote: > >>> >>>> >>>>>>> > > >>> >>>> >>>>>>> > > Hi Bowen, > >>> >>>> >>>>>>> > > > >>> >>>> >>>>>>> > > Thanks for bringing this up. We > >>> actually have > >>> >>>> discussed > >>> >>>> >> about > >>> >>>> >>>>>>> this, and I > >>> >>>> >>>>>>> > > think Till and George have > >>> >>>> >>>>>>> > > already spend sometime investigating > >>> it. I have > >>> >>>> cced both of > >>> >>>> >>>>>>> them, and > >>> >>>> >>>>>>> > > maybe they can share > >>> >>>> >>>>>>> > > their findings. > >>> >>>> >>>>>>> > > > >>> >>>> >>>>>>> > > Best, > >>> >>>> >>>>>>> > > Kurt > >>> >>>> >>>>>>> > > > >>> >>>> >>>>>>> > > > >>> >>>> >>>>>>> > > On Tue, Jun 25, 2019 at 10:08 AM Jark Wu > >>> >>>> <imj...@gmail.com <mailto:imj...@gmail.com> > >>> <mailto:imj...@gmail.com <mailto:imj...@gmail.com>> > >>> >>>> >>>>>>> <mailto:imj...@gmail.com > >>> <mailto:imj...@gmail.com> <mailto:imj...@gmail.com > >>> <mailto:imj...@gmail.com>>>> > >>> >>>> wrote: > >>> >>>> >>>>>>> > > > >>> >>>> >>>>>>> > >> Hi Bowen, > >>> >>>> >>>>>>> > >> > >>> >>>> >>>>>>> > >> Thanks for bringing this. We also > >>> suffered from > >>> >>>> the long > >>> >>>> >>>>>>> build time. > >>> >>>> >>>>>>> > >> I agree that we should focus on > >>> solving build > >>> >>>> capacity > >>> >>>> >>>>>>> problem in the > >>> >>>> >>>>>>> > >> thread. > >>> >>>> >>>>>>> > >> > >>> >>>> >>>>>>> > >> My observation is there is only one > >>> build is > >>> >>>> running, all > >>> >>>> >> the > >>> >>>> >>>>>>> others > >>> >>>> >>>>>>> > >> (other > >>> >>>> >>>>>>> > >> PRs, master) are pending. > >>> >>>> >>>>>>> > >> The pricing plan[1] of travis shows > >>> it can > >>> >>>> support > >>> >>>> >> concurrent > >>> >>>> >>>>>>> build > >>> >>>> >>>>>>> > jobs. > >>> >>>> >>>>>>> > >> But I don't know which plan we are > >>> using, might > >>> >>>> be the free > >>> >>>> >>>>>>> plan for > >>> >>>> >>>>>>> > open > >>> >>>> >>>>>>> > >> source. > >>> >>>> >>>>>>> > >> > >>> >>>> >>>>>>> > >> I cc-ed Chesnay who may have some > >>> experience on > >>> >>>> Travis. > >>> >>>> >>>>>>> > >> > >>> >>>> >>>>>>> > >> Regards, > >>> >>>> >>>>>>> > >> Jark > >>> >>>> >>>>>>> > >> > >>> >>>> >>>>>>> > >> [1]: https://travis-ci.com/plans > >>> >>>> >>>>>>> > >> > >>> >>>> >>>>>>> > >> On Tue, 25 Jun 2019 at 08:11, Bowen Li < > >>> >>>> >> bowenl...@gmail.com <mailto:bowenl...@gmail.com> > >>> <mailto:bowenl...@gmail.com <mailto:bowenl...@gmail.com>> > >>> >>>> >>>>>>> <mailto:bowenl...@gmail.com > >>> <mailto:bowenl...@gmail.com> > >>> >>>> <mailto:bowenl...@gmail.com > >>> <mailto:bowenl...@gmail.com>>>> wrote: > >>> >>>> >>>>>>> > >> > >>> >>>> >>>>>>> > >> > Hi Steven, > >>> >>>> >>>>>>> > >> > > >>> >>>> >>>>>>> > >> > I think you may not read what I > >>> wrote. The > >>> >>>> discussion is > >>> >>>> >>>> about > >>> >>>> >>>>>>> > "unstable > >>> >>>> >>>>>>> > >> > build **capacity**", in another word > >>> >>>> "unstable / lack of > >>> >>>> >>>> build > >>> >>>> >>>>>>> > >> resources", > >>> >>>> >>>>>>> > >> > not "unstable build". > >>> >>>> >>>>>>> > >> > > >>> >>>> >>>>>>> > >> > On Mon, Jun 24, 2019 at 4:40 PM > >>> Steven Wu > >>> >>>> >>>>>>> <stevenz...@gmail.com > >>> <mailto:stevenz...@gmail.com> <mailto:stevenz...@gmail.com > >>> <mailto:stevenz...@gmail.com>> > >>> >>>> <mailto:stevenz...@gmail.com > >>> <mailto:stevenz...@gmail.com> <mailto:stevenz...@gmail.com > >>> <mailto:stevenz...@gmail.com>>>> > >>> >>>> >>>>>>> > wrote: > >>> >>>> >>>>>>> > >> > > >>> >>>> >>>>>>> > >> > > long and sometimes unstable build is > >>> >>>> definitely a pain > >>> >>>> >>>>>> point. > >>> >>>> >>>>>>> > >> > > > >>> >>>> >>>>>>> > >> > > I suspect the build failure here in > >>> >>>> >> flink-connector-kafka > >>> >>>> >>>>>>> is not > >>> >>>> >>>>>>> > >> related > >>> >>>> >>>>>>> > >> > to > >>> >>>> >>>>>>> > >> > > my change. but there is no easy > >>> re-run the > >>> >>>> build on > >>> >>>> >>>>>>> travis UI. > >>> >>>> >>>>>>> > Google > >>> >>>> >>>>>>> > >> > > search showed a trick of > >>> close-and-open the > >>> >>>> PR will > >>> >>>> >>>>>>> trigger rebuild. > >>> >>>> >>>>>>> > >> but > >>> >>>> >>>>>>> > >> > > that could add noises to the PR > >>> activities. > >>> >>>> >>>>>>> > >> > > > >>> >>>> https://travis-ci.org/apache/flink/jobs/545555519 > >>> >>>> >>>>>>> > >> > > > >>> >>>> >>>>>>> > >> > > travis-ci for my personal repo > >>> often failed > >>> >>>> with > >>> >>>> >>>>>>> exceeding time > >>> >>>> >>>>>>> > limit > >>> >>>> >>>>>>> > >> > after > >>> >>>> >>>>>>> > >> > > 4+ hours. > >>> >>>> >>>>>>> > >> > > The job exceeded the maximum time > >>> limit for > >>> >>>> jobs, and > >>> >>>> >> has > >>> >>>> >>>>>>> been > >>> >>>> >>>>>>> > >> > terminated. > >>> >>>> >>>>>>> > >> > > > >>> >>>> >>>>>>> > >> > > On Mon, Jun 24, 2019 at 4:15 PM > >>> Bowen Li > >>> >>>> >>>>>>> <bowenl...@gmail.com > >>> <mailto:bowenl...@gmail.com> <mailto:bowenl...@gmail.com > >>> <mailto:bowenl...@gmail.com>> > >>> >>>> <mailto:bowenl...@gmail.com <mailto:bowenl...@gmail.com> > >>> <mailto:bowenl...@gmail.com <mailto:bowenl...@gmail.com>>>> > >>> >>>> >>>>>>> > wrote: > >>> >>>> >>>>>>> > >> > > > >>> >>>> >>>>>>> > >> > > > > >>> >>>> https://travis-ci.org/apache/flink/builds/549681530 > >>> >>>> >>>>>>> This build > >>> >>>> >>>>>>> > >> > request > >>> >>>> >>>>>>> > >> > > > has > >>> >>>> >>>>>>> > >> > > > been sitting at **HEAD of the > >>> queue** > >>> >>>> since I first > >>> >>>> >> saw > >>> >>>> >>>>>>> it at PST > >>> >>>> >>>>>>> > >> > 10:30am > >>> >>>> >>>>>>> > >> > > > (not sure how long it's been > >>> there before > >>> >>>> 10:30am). > >>> >>>> >>>>>>> It's PST > >>> >>>> >>>>>>> > 4:12pm > >>> >>>> >>>>>>> > >> now > >>> >>>> >>>>>>> > >> > > and > >>> >>>> >>>>>>> > >> > > > it hasn't started yet. > >>> >>>> >>>>>>> > >> > > > > >>> >>>> >>>>>>> > >> > > > On Mon, Jun 24, 2019 at 2:48 PM > >>> Bowen Li > >>> >>>> >>>>>>> <bowenl...@gmail.com > >>> <mailto:bowenl...@gmail.com> <mailto:bowenl...@gmail.com > >>> <mailto:bowenl...@gmail.com>> > >>> >>>> <mailto:bowenl...@gmail.com <mailto:bowenl...@gmail.com> > >>> <mailto:bowenl...@gmail.com <mailto:bowenl...@gmail.com>>>> > >>> >>>> >>>>>>> > >> wrote: > >>> >>>> >>>>>>> > >> > > > > >>> >>>> >>>>>>> > >> > > > > Hi devs, > >>> >>>> >>>>>>> > >> > > > > > >>> >>>> >>>>>>> > >> > > > > I've been experiencing the pain > >>> >>>> resulting from lack > >>> >>>> >>>>>>> of stable > >>> >>>> >>>>>>> > >> build > >>> >>>> >>>>>>> > >> > > > > capacity on Travis for Flink > >>> PRs [1]. > >>> >>>> >> Specifically, I > >>> >>>> >>>>>>> noticed > >>> >>>> >>>>>>> > >> often > >>> >>>> >>>>>>> > >> > > that > >>> >>>> >>>>>>> > >> > > > no > >>> >>>> >>>>>>> > >> > > > > build in the queue is making any > >>> >>>> progress for > >>> >>>> >> hours, > >>> >>>> >>>> and > >>> >>>> >>>>>>> > suddenly > >>> >>>> >>>>>>> > >> 5 > >>> >>>> >>>>>>> > >> > or > >>> >>>> >>>>>>> > >> > > 6 > >>> >>>> >>>>>>> > >> > > > > builds kick off all together > >>> after the > >>> >>>> long pause. > >>> >>>> >>>>>>> I'm at PST > >>> >>>> >>>>>>> > >> > (UTC-08) > >>> >>>> >>>>>>> > >> > > > time > >>> >>>> >>>>>>> > >> > > > > zone, and I've seen pause can > >>> be as > >>> >>>> long as 6 hours > >>> >>>> >>>>>>> from PST 9am > >>> >>>> >>>>>>> > >> to > >>> >>>> >>>>>>> > >> > 3pm > >>> >>>> >>>>>>> > >> > > > > (let alone the time needed to > >>> drain the > >>> >>>> queue > >>> >>>> >>>>>>> afterwards). > >>> >>>> >>>>>>> > >> > > > > > >>> >>>> >>>>>>> > >> > > > > I think this has greatly > >>> impacted our > >>> >>>> productivity. > >>> >>>> >>>> I've > >>> >>>> >>>>>>> > >> experienced > >>> >>>> >>>>>>> > >> > > that > >>> >>>> >>>>>>> > >> > > > > PRs submitted in the early > >>> morning of > >>> >>>> PST time zone > >>> >>>> >>>>>>> won't finish > >>> >>>> >>>>>>> > >> > their > >>> >>>> >>>>>>> > >> > > > > build until late night of the > >>> same day. > >>> >>>> >>>>>>> > >> > > > > > >>> >>>> >>>>>>> > >> > > > > So my questions are: > >>> >>>> >>>>>>> > >> > > > > > >>> >>>> >>>>>>> > >> > > > > - Has anyone else experienced > >>> the same > >>> >>>> problem or > >>> >>>> >>>>>>> have similar > >>> >>>> >>>>>>> > >> > > > observation > >>> >>>> >>>>>>> > >> > > > > on TravisCI? (I suspect it > >>> has things > >>> >>>> to do with > >>> >>>> >> time > >>> >>>> >>>>>>> zone) > >>> >>>> >>>>>>> > >> > > > > > >>> >>>> >>>>>>> > >> > > > > - What pricing plan of > >>> TravisCI is > >>> >>>> Flink currently > >>> >>>> >>>>>>> using? Is it > >>> >>>> >>>>>>> > >> the > >>> >>>> >>>>>>> > >> > > free > >>> >>>> >>>>>>> > >> > > > > plan for open source > >>> projects? What > >>> >>>> are the > >>> >>>> >>>>>>> guaranteed build > >>> >>>> >>>>>>> > >> capacity > >>> >>>> >>>>>>> > >> > > of > >>> >>>> >>>>>>> > >> > > > > the current plan? > >>> >>>> >>>>>>> > >> > > > > > >>> >>>> >>>>>>> > >> > > > > - If the current pricing plan > >>> (either > >>> >>>> free or paid) > >>> >>>> >>>>>> can't > >>> >>>> >>>>>>> > provide > >>> >>>> >>>>>>> > >> > > stable > >>> >>>> >>>>>>> > >> > > > > build capacity, can we > >>> upgrade to a > >>> >>>> higher priced > >>> >>>> >>>>>>> plan with > >>> >>>> >>>>>>> > larger > >>> >>>> >>>>>>> > >> > and > >>> >>>> >>>>>>> > >> > > > more > >>> >>>> >>>>>>> > >> > > > > stable build capacity? > >>> >>>> >>>>>>> > >> > > > > > >>> >>>> >>>>>>> > >> > > > > BTW, another factor that > >>> contribute to > >>> >>>> the > >>> >>>> >>>>>>> productivity problem > >>> >>>> >>>>>>> > is > >>> >>>> >>>>>>> > >> > that > >>> >>>> >>>>>>> > >> > > > > our build is slow - we run > >>> full build > >>> >>>> for every PR > >>> >>>> >>>> and a > >>> >>>> >>>>>>> > >> successful > >>> >>>> >>>>>>> > >> > > full > >>> >>>> >>>>>>> > >> > > > > build takes ~5h. We > >>> definitely have > >>> >>>> more options to > >>> >>>> >>>>>>> solve it, > >>> >>>> >>>>>>> > for > >>> >>>> >>>>>>> > >> > > > instance, > >>> >>>> >>>>>>> > >> > > > > modularize the build graphs > >>> and reuse > >>> >>>> artifacts > >>> >>>> >> from > >>> >>>> >>>> the > >>> >>>> >>>>>>> > previous > >>> >>>> >>>>>>> > >> > > build. > >>> >>>> >>>>>>> > >> > > > > But I think that can be a big > >>> effort > >>> >>>> which is much > >>> >>>> >>>>>>> harder to > >>> >>>> >>>>>>> > >> > accomplish > >>> >>>> >>>>>>> > >> > > > in > >>> >>>> >>>>>>> > >> > > > > a short period of time and > >>> may deserve > >>> >>>> its own > >>> >>>> >>>> separate > >>> >>>> >>>>>>> > >> discussion. > >>> >>>> >>>>>>> > >> > > > > > >>> >>>> >>>>>>> > >> > > > > [1] > >>> >>>> >> https://travis-ci.org/apache/flink/pull_requests > >>> >>>> >>>>>>> > >> > > > > > >>> >>>> >>>>>>> > >> > > > > > >>> >>>> >>>>>>> > >> > > > > >>> >>>> >>>>>>> > >> > > > >>> >>>> >>>>>>> > >> > > >>> >>>> >>>>>>> > >> > >>> >>>> >>>>>>> > > > >>> >>>> >>>>>>> > > >>> >>>> >>>>>>> > >>> >>>> >>>>>>> > >>> >>>> >>>>>>> -- > >>> >>>> >>>>>>> Best Regards > >>> >>>> >>>>>>> > >>> >>>> >>>>>>> Jeff Zhang > >>> >>>> >>>>>>> > >>> >>>> >> > >>> >>>> > >>> >>> > >>> >> > >>> > >> > >> > > > >
