On Thu, Apr 14, 2016 at 10:36 AM, Jake Farrell <jfarr...@apache.org> wrote:
> completely agree, without passing CI we have no gauge on what impact a > patch could make and no guarantee that it will not destabilize the project. > I am a +1 to holding the 0.10.0 release candidate until we can get CI back > to green and kept that way > I'm very glad to hear this! I think I can be hands-off on the Jenkins Thrift-precommit job now. https://builds.apache.org/job/Thrift-precommit/418 is running with changes that include the equivalent of `git clean -fdx && git reset --hard HEAD` to get around the merge issue as well as more shimming in the docker image build to add a jenkins user to the image to avoid perms issues. > -Jake > > On Thu, Apr 14, 2016 at 11:55 AM, John Sirois <jsir...@apache.org> wrote: > >> >> >> On Thu, Apr 14, 2016 at 9:52 AM, Jake Farrell <jfarr...@apache.org> >> wrote: >> >>> +1 to reducing this as much as possible so there is less maintenance >>> overhead and setup. Ideally i would love to see all this dockerized so we >>> are using a common base across the board and then either travis or jenkins >>> or locally can all run those containers with the same setup in docker and >>> the same outlined test scripts (which should be within the build folder) so >>> this is the same repeatable process where ever it is run from >>> >> >> I said this on another thread, but I think your and other comitter >> limited resources should be bent towards doing this now and not accepting >> patches, making changes or doing RCs. >> >> The thrift project feels untenable to contribute to with all the red CI. >> After seeing this, I'm also growing disinclined towards depending on the >> tool in my projects. >> Maybe the thrift committers don't share this sense of priority, but to me >> a stable green CI is the foundation of any project, without it everything >> else crumbles. >> >> >>> >>> >>> >>> On Thu, Apr 14, 2016 at 11:41 AM, John Sirois <jsir...@apache.org> >>> wrote: >>> >>>> On Thu, Apr 14, 2016 at 9:34 AM, Aki Sukegawa <ns...@apache.org> wrote: >>>> >>>> > Quoting from my previous mail. >>>> > >>>> > > Other than Travis, make check is hanging for almost every build of >>>> > Jenkins. >>>> > > The log is not that clear but I think it's D test. >>>> > > AFAIK the test was running fine a few weeks ago and nobody touched >>>> it >>>> > since then. >>>> > > It might be due to insufficient resource on Jenkins. >>>> > >>>> > I suspect default task limit introduced in a recent version of docker >>>> is >>>> > not lifted on ASF jenkins. >>>> > >>>> > I'm not sure if it's worth maintaining sub-set of builds on another CI >>>> > that has relatively unstable basis that cannot even be touched by >>>> > committers. >>>> > Less resource is fine because it can detect failures on such platforms >>>> > like last time John enabled it. >>>> > But it's apparently changing. >>>> > >>>> >>>> Aha - that would be an interesting cause to the D hangs. >>>> >>>> I'm not clear on what you meant by the rest, but I assume you're >>>> addressing >>>> the confusing fact that thrift maintains 2 sets of broken CI jobs >>>> (fwict) >>>> for pull requests, TravisCI and Apache Jenkins. >>>> >>>> It seems to me 4 steps are needed to provide baseline sanity for >>>> contributing to the project: >>>> 1. Halt accepting and changes immediately. >>>> 2. Pick Travis or Jenkins, kill the other. >>>> 3. Get the winner from 2 green. >>>> 4. Resume accepting patches that are green in CI and only green in CI. >>>> >>>> >>>> > On Thu, Apr 14, 2016 at 11:45 PM John Sirois <jsir...@apache.org> >>>> wrote: >>>> > >>>> >> On Thu, Apr 14, 2016 at 8:29 AM, John Sirois <jsir...@apache.org> >>>> wrote: >>>> >> >>>> >> > >>>> >> > >>>> >> > On Thu, Apr 14, 2016 at 8:14 AM, Jim King <jim.k...@simplivity.com >>>> > >>>> >> wrote: >>>> >> > >>>> >> >> I got one build through (which failed in "d" tests) and now it's >>>> stuck >>>> >> in >>>> >> >> the same state, see: >>>> >> >> https://builds.apache.org/job/Thrift-precommit/411/ >>>> >> >> >>>> >> >> FATAL: Could not checkout master with start point origin/master >>>> >> >> hudson.plugins.git.GitException: Could not checkout master with >>>> start >>>> >> >> point origin/master >>>> >> >> at >>>> >> >> >>>> >> >>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:1962) >>>> >> >> at >>>> >> >> >>>> >> >>>> org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.checkoutBranch(AbstractGitAPIImpl.java:82) >>>> >> >> at >>>> >> >> >>>> >> >>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.checkoutBranch(CliGitAPIImpl.java:62) >>>> >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >>>> Method) >>>> >> >> at >>>> >> >> >>>> >> >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>> >> >> at >>>> >> >> >>>> >> >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> >> >> at java.lang.reflect.Method.invoke(Method.java:606) >>>> >> >> at >>>> >> >> >>>> >> >>>> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:608) >>>> >> >> at >>>> >> >> >>>> >> >>>> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583) >>>> >> >> at >>>> >> >> >>>> >> >>>> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542) >>>> >> >> at >>>> hudson.remoting.UserRequest.perform(UserRequest.java:120) >>>> >> >> at >>>> hudson.remoting.UserRequest.perform(UserRequest.java:48) >>>> >> >> at hudson.remoting.Request$2.run(Request.java:326) >>>> >> >> at >>>> >> >> >>>> >> >>>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) >>>> >> >> at >>>> java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>> >> >> at >>>> >> >> >>>> >> >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> >> >> at >>>> >> >> >>>> >> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> >> >> at java.lang.Thread.run(Thread.java:745) >>>> >> >> at ......remote call to H10(Native Method) >>>> >> >> at >>>> >> >> >>>> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416) >>>> >> >> at >>>> hudson.remoting.UserResponse.retrieve(UserRequest.java:220) >>>> >> >> at hudson.remoting.Channel.call(Channel.java:781) >>>> >> >> at >>>> >> >> >>>> >> >>>> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250) >>>> >> >> at com.sun.proxy.$Proxy115.checkoutBranch(Unknown Source) >>>> >> >> at >>>> >> >> >>>> >> >>>> org.jenkinsci.plugins.gitclient.RemoteGitImpl.checkoutBranch(RemoteGitImpl.java:327) >>>> >> >> at >>>> >> >> >>>> >> >>>> com.cloudbees.jenkins.plugins.git.vmerge.BuildChooserImpl.getCandidateRevisions(BuildChooserImpl.java:78) >>>> >> >> at >>>> >> >> >>>> hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:951) >>>> >> >> at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1054) >>>> >> >> at hudson.scm.SCM.checkout(SCM.java:485) >>>> >> >> at >>>> >> >> hudson.model.AbstractProject.checkout(AbstractProject.java:1276) >>>> >> >> at >>>> >> >> >>>> >> >>>> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607) >>>> >> >> at >>>> >> >> >>>> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) >>>> >> >> at >>>> >> >> >>>> >> >>>> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529) >>>> >> >> at hudson.model.Run.execute(Run.java:1738) >>>> >> >> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) >>>> >> >> at >>>> >> >> >>>> hudson.model.ResourceController.execute(ResourceController.java:98) >>>> >> >> at hudson.model.Executor.run(Executor.java:410) >>>> >> >> Caused by: hudson.plugins.git.GitException: Command "git checkout >>>> -b >>>> >> >> master origin/master" returned status code 1: >>>> >> >> stdout: lib/lua/TCompactProtocol.lua: needs merge >>>> >> >> >>>> >> >> stderr: error: you need to resolve your current index first >>>> >> >> >>>> >> >> It looks like the build environment is not forced clean at the >>>> >> beginning >>>> >> >> of each build. >>>> >> >> >>>> >> > >>>> >> > Ack - looking now. >>>> >> > >>>> >> > It is odd that the git portion of these builds went sideways since >>>> the >>>> >> > Jenkins Job Config History auditing plugin shows the last change >>>> >> (before my >>>> >> > tweak last night) was 2016-02-16_02-09-39. I expect jenkins or its >>>> >> plugins >>>> >> > were updated by infra causing the previously working job config to >>>> not >>>> >> work >>>> >> > any longer. >>>> >> > >>>> >> >>>> >> OK - that analysis was wrong, clearly there has been a change in the >>>> build >>>> >> itself that modifies source code and this causes the issue. >>>> >> I've enabled >>>> <hudson.plugins.git.extensions.impl.CleanBeforeCheckout/> >>>> >> with >>>> >> the following description: >>>> >> >>>> >> Clean up the workspace before every checkout by deleting all >>>> untracked >>>> >> files and directories, including those which are specified in >>>> .gitignore. >>>> >> It also resets all *tracked* files to their versioned state. This >>>> ensures >>>> >> >>>> >> that the workspace is in the same state as if you cloned and checked >>>> out >>>> >> in >>>> >> a brand-new empty directory, and ensures that your build is not >>>> affected >>>> >> by >>>> >> the files generated by the previous build. >>>> >> >>>> >> That sounds like ~ `git clean -fdx && git reset --hard HEAD` to me, >>>> which >>>> >> should do it. That should insulate CI from bad tests that modify >>>> checked >>>> >> in >>>> >> repo state, but those tests shouldn't exist either. >>>> >> >>>> >> COMMITTERS: >>>> >> I'd like to reiterate to any committers out there that red CI must >>>> be a >>>> >> hard bright line that is not crossed when accepting patches; >>>> otherwise >>>> >> well >>>> >> be right back here after getting this thing green again. Here we is >>>> you - >>>> >> I won't be interested in helping out a third time if this relapses. >>>> >> >>>> >> >>>> >> > >>>> >> >> - Jim >>>> >> >> >>>> >> >> -----Original Message----- >>>> >> >> From: Jim King >>>> >> >> Sent: Wednesday, April 13, 2016 10:34 PM >>>> >> >> To: dev@thrift.apache.org; 'jsir...@apache.org' < >>>> jsir...@apache.org> >>>> >> >> Subject: RE: Build Failures >>>> >> >> >>>> >> >> The builds were failing claiming that a file was in the middle of >>>> being >>>> >> >> merged and they were all failing early. >>>> >> >> I think the build environment itself is compromised and there's >>>> >> nothing I >>>> >> >> can do about that. >>>> >> >> >>>> >> >> -----Original Message----- >>>> >> >> From: John Sirois [mailto:jsir...@apache.org] >>>> >> >> Sent: Wednesday, April 13, 2016 9:58 PM >>>> >> >> To: dev@thrift.apache.org >>>> >> >> Subject: Re: Build Failures >>>> >> >> >>>> >> >> On Wed, Apr 13, 2016 at 7:54 PM, John Sirois <jsir...@apache.org> >>>> >> wrote: >>>> >> >> >>>> >> >> > >>>> >> >> > >>>> >> >> > On Wed, Apr 13, 2016 at 7:51 PM, Jim King < >>>> jim.k...@simplivity.com> >>>> >> >> wrote: >>>> >> >> > >>>> >> >> >> I’m still looking for answers on pull request build failures. >>>> >> >> >> >>>> >> >> >> I have 2 or 3 PRs open right now and they’ve failed in the >>>> apache >>>> >> >> >> precommit builds for strange reasons. >>>> >> >> >> >>>> >> >> >> The apache internal builds seem to be failing. >>>> >> >> >> >>>> >> >> > >>>> >> >> > I think the answer is the breaks need a fixer; hopefully you >>>> can find >>>> >> >> > time to help fix. >>>> >> >> > >>>> >> >> > I say this because I started down a series of patches to the >>>> java >>>> >> >> > codegen/lib a while back and found a similar state - though on >>>> the >>>> >> >> > pull request builder (apache jenkins). I stopped my java stuff >>>> and >>>> >> >> > fixed that CI with the help of Aki and Jake reviewing and >>>> providing >>>> >> >> > guidance. I am not a thrift comitter. >>>> >> >> > >>>> >> >> >>>> >> >> I will say that its discouraging that that CI is now solid red >>>> too: >>>> >> >> https://builds.apache.org/job/Thrift-precommit/ >>>> >> >> Part of the answer IMO is for committers to hold a hard line on >>>> >> accepting >>>> >> >> any patch, or pushing their own, w/o full green CIs. >>>> >> >> >>>> >> >> >>>> >> >> > >>>> >> >> > >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> [image: Description: Description: simplivity-lg-xsmall] >>>> >> >> >> >>>> >> >> >> James E. King, III >>>> >> >> >> >>>> >> >> >> Architect >>>> >> >> >> >>>> >> >> >> 8 Technology Drive, 2nd Floor >>>> >> >> >> Westborough, MA 01581-1756 >>>> >> >> >> >>>> >> >> >> Ph: 855-SVT-INFO >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> ------------------------------ >>>> >> >> >> PRIVACY STATEMENT: >>>> >> >> >> This message is a PRIVATE communication. This message and all >>>> >> >> >> attachments are a private communication sent by SimpliVity and >>>> are >>>> >> >> >> considered to be confidential or protected by privilege. If >>>> you are >>>> >> >> >> not the intended recipient, you are hereby notified that any >>>> >> >> >> disclosure, copying, distribution or use of the information >>>> >> contained >>>> >> >> >> in or attached to this message is strictly prohibited. Please >>>> notify >>>> >> >> >> the sender of the delivery error by replying to this message, >>>> and >>>> >> then >>>> >> >> delete it from your system. >>>> >> >> >> For more information please visit http://www.simplivity.com >>>> >> >> >> ------------------------------ >>>> >> >> >> >>>> >> >> > >>>> >> >> > >>>> >> >> >>>> >> >> >>>> >> >>>> --------------------------------------------------------------------------------------- >>>> >> >> PRIVACY STATEMENT: >>>> >> >> This message is a PRIVATE communication. This message and all >>>> >> >> attachments are a private communication sent by SimpliVity and are >>>> >> >> considered to be confidential or protected by privilege. If you >>>> are >>>> >> not the >>>> >> >> intended recipient, you are hereby notified that any disclosure, >>>> >> copying, >>>> >> >> distribution or use of the information contained in or attached >>>> to this >>>> >> >> message is strictly prohibited. Please notify the sender of the >>>> >> delivery >>>> >> >> error by replying to this message, and then delete it from your >>>> system. >>>> >> >> >>>> >> >> >>>> >> >>>> --------------------------------------------------------------------------------------- >>>> >> >> >>>> >> > >>>> >> > >>>> >> >>>> > >>>> >>> >>> >> >
