Hi Pavol, > On May 11, 2017, at 1:33 PM, Pavol Vaskovic <p...@pali.sk> wrote: > > Hi Mishal, > > Douglas Gregor suggested in SR-4814 (that was merged as dupe into SR-4669) > that I get in contact with you because I’m working on issue SR-4669 Add a > Benchmark_Driver --rerun N option <https://bugs.swift.org/browse/SR-4669> and > I should make sure that I don’t break the Swift-CI. > > The point of that is to deal with false changes (regressions and > improvements) that are often seen in benchmark reports from CI. > > I have run Benchmark_O with --num-iters=100 on my machine for the the whole > performance test suite, to get a feeling for the distribution of benchmark > samples, because I also want to move the Benchmark_Driver to use MEAN instead > of MIN in the analysis. > > There was a tremendous difference on the quality of measured samples > depending on how busy my machine was. When I had idle browser windows open in > the background, or Xcode open, the results had very high variance. When I > purged all I could the results were very stable. > > First I want to check with you that the Macs that run the CI and benchmarks > do have turned off the Spotlight indexing on the folder that they check out > the project into > (/Users/buildnode/jenkins/workspace/swift-PR-osx-perf/swift). Just making > sure that `mds`, `mds_stores` and `mds_worker` processes don’t unnecessarily > impact the benchmark runs. How about Time Machine's `backupd`? Yes, we have these services disabled.
> > --- > > Also I have recently filed SR-4659 Benchmark logs should be tied to tested > tree version <https://bugs.swift.org/browse/SR-4659> and implemented a fix > that got merged in PR #8978 <https://github.com/apple/swift/pull/8978>, > adding short hash of the commit to the Benchmark log file name. But that > works only while you don’t make local changes. In the end I would like to > have fully identified versions of benchmarks in the comparison report. In > order to make that work, I’ve filed SR-4675 Report detailed build version > from Benchmark drivers <https://bugs.swift.org/browse/SR-4675>, that calls > for having something akin to `swift —version` be baked into the > Benchmark_O{Onone, Ounchecked}. > > Given we are on git now, I think the solution is to modify cmake to run `git > describe` and generate some kind of version file that will be included during > build and reported by the driver when running benchmarks. > > I thought it would be nice touch to report also the branch names like one > that is cited in the PRs on github. For example: > > apple:master vs palimondo:SR-4659 For pull request jobs following env will be available. ${ghprbTargetBranch} (master) vs ${ghprbSourceBranch} (SR-4659) > > I found that getSVN.cmake > <https://github.com/apple/swift-llvm/blob/stable/cmake/modules/GetSVN.cmake#L73> > from llvm has all the commands to extract the needed information… But, whenI > looked at full console logs from swift-ci to see if that would work with the > way how the branches are checked out on the build machine, I’m a bit confused: > >> Cloning the remote Git repository >> Cloning repository g...@github.com <mailto:g...@github.com>:apple/swift.git >> > git init /Users/buildnode/jenkins/workspace/swift-PR-osx-perf/swift # >> timeout=10 >> Fetching upstream changes from g...@github.com >> <mailto:g...@github.com>:apple/swift.git >> > git --version # timeout=10 >> > git fetch --tags --progress g...@github.com >> <mailto:g...@github.com>:apple/swift.git +refs/heads/*:refs/remotes/origin/* >> > git config remote.origin.url g...@github.com >> <mailto:g...@github.com>:apple/swift.git # timeout=10 >> > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* >> # timeout=10 >> > git config remote.origin.url g...@github.com >> <mailto:g...@github.com>:apple/swift.git # timeout=10 >> Cleaning workspace >> > git rev-parse --verify HEAD # timeout=10 >> No valid HEAD. Skipping the resetting >> > git clean -fdx # timeout=10 >> Fetching upstream changes from g...@github.com >> <mailto:g...@github.com>:apple/swift.git >> > git fetch --tags --progress g...@github.com >> <mailto:g...@github.com>:apple/swift.git >> +refs/pull/*:refs/remotes/origin/pr/* >> > git rev-parse refs/remotes/origin/pr/9330/merge^{commit} # timeout=10 >> > git rev-parse refs/remotes/origin/origin/pr/9330/merge^{commit} # >> timeout=10 >> Checking out Revision 55f2b7e24356b31fd29c690d779f9aaf4a3d59d5 >> (refs/remotes/origin/pr/9330/merge) >> > git config core.sparsecheckout # timeout=10 >> > git checkout -f 55f2b7e24356b31fd29c690d779f9aaf4a3d59d5 >> Using 'Changelog to branch' strategy. >> [swift-PR-osx-perf] $ /bin/bash -e >> /var/folders/_8/79jmzf2142z2xydc_01btlx00000gn/T/hudson7057602820198606732.sh >> Cloning into 'ninja'... >> Branch: master >> Found related pull requests: [] > > > I don’t quite understand how is the PR branch cloned from github. It > definitely looks different from the situation wehn I build on my local > machine. Github provides merge ref for PR, for example "origin/pr/9330/merge”. This allows us to test PR with latest changes from targeted branch, avoiding out of date PR testing. > > What would be the way to extract remote repository and branch name for the PR > being merged? You can use ${ghprbTargetBranch} for targeted branch, ${ghprbSourceBranch} for PR branch, ${ghprbPullAuthorLogin} for PR author. Full list of env: https://ci.swift.org/view/Pull%20Request/job/swift-PR-osx-perf/558/parameters/ > > When I’ll be working on these changes, It would be able to build just the > benchmark suite, but SR-4593 Following instructions to independently build > benchmarks fails to do so <https://bugs.swift.org/browse/SR-4593>. Who can > help me here? (So far I have always been using ./swift/utils/build-script -R, > but that always takes ages, and often rebuilds whole compiler infrastructure.) > > Best regards > Pavol Vaskovic > >
_______________________________________________ swift-dev mailing list swift-dev@swift.org https://lists.swift.org/mailman/listinfo/swift-dev
