Re: [julia-users] Test code taking forever to start
I did a little bit more bisecting to try and find the most recent regression, but I'm not sure if I nailed it or not. All the data is now summarised in the ticket here: https://github.com/JuliaLang/julia/issues/15615
Re: [julia-users] Test code taking forever to start
Rather than trying to nail down the remaining regressions, I think I am just going to give the data I've collected, which illustrates a problem, namely that in a very short space there can be massive improvements, immediately followed by massive regressions. This essentially violates the git bisect working assumption that there is one bad commit and you are trying to find it. Here is a list of commits I have hit during various commits and the times I got for those commits. There was also another commit I hit where the time improved by a factor of 2, but all subsequent attempts to build any commits of Julia failed and so I had to clone the repository again. Because of this, I threw that data point away as likely corrupt. 41fb1ba good 15s 79a0d7b good 5ea20fc good 275c7e8 good bea07fc good c5704ed good 0318444 good 6396218 bad1 3b7f18a bad1 63daf4f bad1 1bfabbb bad1 179a439 bad1 0b6cab6 bad1 d8ec4a7 bad1 064b03c bad1 53b02a6 bad1 ca6f253 bad1 42s 8f4238a bad2 59s f67203c bad2 53s af81431 bad2 58s 6382116 bad3 73s 24276a5 bad3 74s 79b08ca bad3 73s 93cb2ae bad3 71s 5baedf4 bad3 72s 70a3120 ?? 48s 8af0134 bad4 82s def50a0 bad4 78s 039f57b bad4 78s 597dc7b bad4 78s adc1ed8 bad4 78s fc469b6 bad4 78s be9f208 bad5 87s master bad5 90s In summary, the commits that I know are serious regressions for sure are: 6396218 8f4238a 6382116 8af0134 There's also a regression somewhere between fc469b6 and cf68f4d3 (the latter being master some time today) which I didn't bisect yet. At this point I'm not sure how useful this is. Bill.
Re: [julia-users] Test code taking forever to start
Thanks for taking the time and effort to track these down, Bill. We should try to distill some tests for each of these regressions so that we can guard against performance regressions for the same sorts of tasks in the future. On Thu, Mar 24, 2016 at 9:20 AM, 'Bill Hart' via julia-users < julia-users@googlegroups.com> wrote: > The third regression is: > > 6382116 Jameson Nash implement recursive > type-inference > > It's about a 25% regression. There are some other much smaller regressions > (~7%) before that which I skipped over. > > Still trying to track down a potential fourth regression. > > Bill. >
Re: [julia-users] Test code taking forever to start
The third regression is: 6382116 Jameson Nash implement recursive type-inference It's about a 25% regression. There are some other much smaller regressions (~7%) before that which I skipped over. Still trying to track down a potential fourth regression. Bill.
Re: [julia-users] Test code taking forever to start
These should imo all be on the issue tracker. AFAIU most of these commits where supposed to speed up type inference. On Thursday, March 24, 2016 at 12:46:07 PM UTC+1, Bill Hart wrote: > > I'm sorry, please ignore that merge commit. I think that git bisect > screwed up due to the fact that there are in fact a couple of major > regressions in the range I was bisecting. > > The first of these is at the following commit: > > 8f4238a do type-inference using a work queue > > It's about a 35% regression. > > There are still at least two regressions to track down. > > Bill. >
Re: [julia-users] Test code taking forever to start
I'm sorry, please ignore that merge commit. I think that git bisect screwed up due to the fact that there are in fact a couple of major regressions in the range I was bisecting. The first of these is at the following commit: 8f4238a do type-inference using a work queue It's about a 35% regression. There are still at least two regressions to track down. Bill.
Re: [julia-users] Test code taking forever to start
The second bad commit is: 7facd20 Merge pull request #15300 from JuliaLang/jn/typeinfq However this is only about a 60% regression. There is still a third commit that is bad. I will try to bisect it today if I can. There is also a fourth smaller regression of about 10% somewhere, but I didn't bother tracking that down as we can live with that. Bill. On Thursday, 24 March 2016 02:15:57 UTC+1, Tim Holy wrote: > > That's a big help. I'm not certain Jeff reads julia-users, however, so you > might want to file an issue. > > Best, > --Tim > > On Wednesday, March 23, 2016 04:34:25 PM 'Bill Hart' via julia-users > wrote: > > On Thursday, 24 March 2016 00:33:04 UTC+1, Bill Hart wrote: > > > It seems there may have been two regressions. The first regression > with a > > > slowdown factor of just over 2 seems to be: > > > > > > 639621859863609c5f3abbc2ed75c675695b3693 is the first bad commit > > > commit 639621859863609c5f3abbc2ed75c675695b3693 > > > Author: Jeff Bezanson > > > > Date: Tue Jan 26 23:33:19 2016 -0500 > > > > > > modify Base.Test not to create a closure for each test > > > : > > > :04 04 9c84c85afaed99190f3e744123dccc732f2c760e > > > > > > 486795536d95d1fb14fd9f7f415fb63cd9c6e490 M base > > > > > > :04 04 48205a7b1b007692c81b1a8d931cb44f6cc97be8 > > > > > > acb43cd0ecece4237e1834b7a3b577f312884650 M test > > > > > > I will try to find time to find the second regression, which occurs > > > between 1bfabbb and 1bfabbb I believe. > > > > Sorry, that should say between 1bfabbb and 9d6e726. > > > > Bill. > > > > > On Wednesday, 23 March 2016 15:23:05 UTC+1, Bill Hart wrote: > > >> I'll see if it is possible. Currently our code does not work at all > with > > >> large chunks of the Julia commits in that interval. We had to work > around > > >> various things and don't know precisely when they were switched on or > > >> off. > > >> > > >> Bill. > > >> > > >> On Wednesday, 23 March 2016 14:54:22 UTC+1, Tim Holy wrote: > > >>> If you can git-bisect the change, it would be a huge help. > > >>> > > >>> Best, > > >>> --Tim > > >>> > > >>> On Wednesday, March 23, 2016 06:18:23 AM 'Bill Hart' via julia-users > > >>> > > >>> wrote: > > >>> > In very recent Julia-0.5-dev the test code in our Nemo module > takes > > >>> > > >>> forever > > >>> > > >>> > to start running. It's close to 2 minutes. > > >>> > > > >>> > This compares with about 15s with older Julia-0.5-dev, say 3 > months > > >>> > > >>> ago > > >>> > > >>> > before the LLVM switchover. > > >>> > > > >>> > Does anyone know why there is this massive performance regression. > Is > > >>> > > >>> it > > >>> > > >>> > likely that it can be fixed? It's really killing our development > > >>> > > >>> cycle. > > >>> > > >>> > Bill. > >
Re: [julia-users] Test code taking forever to start
Oh wow, I didn't even know that commit existed. Now BaseTestNext.jl and Base.Test are out of sync... Bill, can you try using BaseTestNext.jl instead of Base.Test? If this change was indeed responsible, then that might reveal it. On Wednesday, March 23, 2016 at 9:15:57 PM UTC-4, Tim Holy wrote: > > That's a big help. I'm not certain Jeff reads julia-users, however, so you > might want to file an issue. > > Best, > --Tim > > On Wednesday, March 23, 2016 04:34:25 PM 'Bill Hart' via julia-users > wrote: > > On Thursday, 24 March 2016 00:33:04 UTC+1, Bill Hart wrote: > > > It seems there may have been two regressions. The first regression > with a > > > slowdown factor of just over 2 seems to be: > > > > > > 639621859863609c5f3abbc2ed75c675695b3693 is the first bad commit > > > commit 639621859863609c5f3abbc2ed75c675695b3693 > > > Author: Jeff Bezanson > > > > Date: Tue Jan 26 23:33:19 2016 -0500 > > > > > > modify Base.Test not to create a closure for each test > > > : > > > :04 04 9c84c85afaed99190f3e744123dccc732f2c760e > > > > > > 486795536d95d1fb14fd9f7f415fb63cd9c6e490 M base > > > > > > :04 04 48205a7b1b007692c81b1a8d931cb44f6cc97be8 > > > > > > acb43cd0ecece4237e1834b7a3b577f312884650 M test > > > > > > I will try to find time to find the second regression, which occurs > > > between 1bfabbb and 1bfabbb I believe. > > > > Sorry, that should say between 1bfabbb and 9d6e726. > > > > Bill. > > > > > On Wednesday, 23 March 2016 15:23:05 UTC+1, Bill Hart wrote: > > >> I'll see if it is possible. Currently our code does not work at all > with > > >> large chunks of the Julia commits in that interval. We had to work > around > > >> various things and don't know precisely when they were switched on or > > >> off. > > >> > > >> Bill. > > >> > > >> On Wednesday, 23 March 2016 14:54:22 UTC+1, Tim Holy wrote: > > >>> If you can git-bisect the change, it would be a huge help. > > >>> > > >>> Best, > > >>> --Tim > > >>> > > >>> On Wednesday, March 23, 2016 06:18:23 AM 'Bill Hart' via julia-users > > >>> > > >>> wrote: > > >>> > In very recent Julia-0.5-dev the test code in our Nemo module > takes > > >>> > > >>> forever > > >>> > > >>> > to start running. It's close to 2 minutes. > > >>> > > > >>> > This compares with about 15s with older Julia-0.5-dev, say 3 > months > > >>> > > >>> ago > > >>> > > >>> > before the LLVM switchover. > > >>> > > > >>> > Does anyone know why there is this massive performance regression. > Is > > >>> > > >>> it > > >>> > > >>> > likely that it can be fixed? It's really killing our development > > >>> > > >>> cycle. > > >>> > > >>> > Bill. > >
Re: [julia-users] Test code taking forever to start
That's a big help. I'm not certain Jeff reads julia-users, however, so you might want to file an issue. Best, --Tim On Wednesday, March 23, 2016 04:34:25 PM 'Bill Hart' via julia-users wrote: > On Thursday, 24 March 2016 00:33:04 UTC+1, Bill Hart wrote: > > It seems there may have been two regressions. The first regression with a > > slowdown factor of just over 2 seems to be: > > > > 639621859863609c5f3abbc2ed75c675695b3693 is the first bad commit > > commit 639621859863609c5f3abbc2ed75c675695b3693 > > Author: Jeff Bezanson > > Date: Tue Jan 26 23:33:19 2016 -0500 > > > > modify Base.Test not to create a closure for each test > > : > > :04 04 9c84c85afaed99190f3e744123dccc732f2c760e > > > > 486795536d95d1fb14fd9f7f415fb63cd9c6e490 M base > > > > :04 04 48205a7b1b007692c81b1a8d931cb44f6cc97be8 > > > > acb43cd0ecece4237e1834b7a3b577f312884650 M test > > > > I will try to find time to find the second regression, which occurs > > between 1bfabbb and 1bfabbb I believe. > > Sorry, that should say between 1bfabbb and 9d6e726. > > Bill. > > > On Wednesday, 23 March 2016 15:23:05 UTC+1, Bill Hart wrote: > >> I'll see if it is possible. Currently our code does not work at all with > >> large chunks of the Julia commits in that interval. We had to work around > >> various things and don't know precisely when they were switched on or > >> off. > >> > >> Bill. > >> > >> On Wednesday, 23 March 2016 14:54:22 UTC+1, Tim Holy wrote: > >>> If you can git-bisect the change, it would be a huge help. > >>> > >>> Best, > >>> --Tim > >>> > >>> On Wednesday, March 23, 2016 06:18:23 AM 'Bill Hart' via julia-users > >>> > >>> wrote: > >>> > In very recent Julia-0.5-dev the test code in our Nemo module takes > >>> > >>> forever > >>> > >>> > to start running. It's close to 2 minutes. > >>> > > >>> > This compares with about 15s with older Julia-0.5-dev, say 3 months > >>> > >>> ago > >>> > >>> > before the LLVM switchover. > >>> > > >>> > Does anyone know why there is this massive performance regression. Is > >>> > >>> it > >>> > >>> > likely that it can be fixed? It's really killing our development > >>> > >>> cycle. > >>> > >>> > Bill.
Re: [julia-users] Test code taking forever to start
On Thursday, 24 March 2016 00:33:04 UTC+1, Bill Hart wrote: > > It seems there may have been two regressions. The first regression with a > slowdown factor of just over 2 seems to be: > > 639621859863609c5f3abbc2ed75c675695b3693 is the first bad commit > commit 639621859863609c5f3abbc2ed75c675695b3693 > Author: Jeff Bezanson > Date: Tue Jan 26 23:33:19 2016 -0500 > > modify Base.Test not to create a closure for each test > > :04 04 9c84c85afaed99190f3e744123dccc732f2c760e > 486795536d95d1fb14fd9f7f415fb63cd9c6e490 M base > :04 04 48205a7b1b007692c81b1a8d931cb44f6cc97be8 > acb43cd0ecece4237e1834b7a3b577f312884650 M test > > I will try to find time to find the second regression, which occurs > between 1bfabbb and 1bfabbb I believe. > > Sorry, that should say between 1bfabbb and 9d6e726. Bill. > > On Wednesday, 23 March 2016 15:23:05 UTC+1, Bill Hart wrote: >> >> I'll see if it is possible. Currently our code does not work at all with >> large chunks of the Julia commits in that interval. We had to work around >> various things and don't know precisely when they were switched on or off. >> >> Bill. >> >> On Wednesday, 23 March 2016 14:54:22 UTC+1, Tim Holy wrote: >>> >>> If you can git-bisect the change, it would be a huge help. >>> >>> Best, >>> --Tim >>> >>> On Wednesday, March 23, 2016 06:18:23 AM 'Bill Hart' via julia-users >>> wrote: >>> > In very recent Julia-0.5-dev the test code in our Nemo module takes >>> forever >>> > to start running. It's close to 2 minutes. >>> > >>> > This compares with about 15s with older Julia-0.5-dev, say 3 months >>> ago >>> > before the LLVM switchover. >>> > >>> > Does anyone know why there is this massive performance regression. Is >>> it >>> > likely that it can be fixed? It's really killing our development >>> cycle. >>> > >>> > Bill. >>> >>>
Re: [julia-users] Test code taking forever to start
It seems there may have been two regressions. The first regression with a slowdown factor of just over 2 seems to be: 639621859863609c5f3abbc2ed75c675695b3693 is the first bad commit commit 639621859863609c5f3abbc2ed75c675695b3693 Author: Jeff Bezanson Date: Tue Jan 26 23:33:19 2016 -0500 modify Base.Test not to create a closure for each test :04 04 9c84c85afaed99190f3e744123dccc732f2c760e 486795536d95d1fb14fd9f7f415fb63cd9c6e490 M base :04 04 48205a7b1b007692c81b1a8d931cb44f6cc97be8 acb43cd0ecece4237e1834b7a3b577f312884650 M test I will try to find time to find the second regression, which occurs between 1bfabbb and 1bfabbb I believe. Bill. On Wednesday, 23 March 2016 15:23:05 UTC+1, Bill Hart wrote: > > I'll see if it is possible. Currently our code does not work at all with > large chunks of the Julia commits in that interval. We had to work around > various things and don't know precisely when they were switched on or off. > > Bill. > > On Wednesday, 23 March 2016 14:54:22 UTC+1, Tim Holy wrote: >> >> If you can git-bisect the change, it would be a huge help. >> >> Best, >> --Tim >> >> On Wednesday, March 23, 2016 06:18:23 AM 'Bill Hart' via julia-users >> wrote: >> > In very recent Julia-0.5-dev the test code in our Nemo module takes >> forever >> > to start running. It's close to 2 minutes. >> > >> > This compares with about 15s with older Julia-0.5-dev, say 3 months ago >> > before the LLVM switchover. >> > >> > Does anyone know why there is this massive performance regression. Is >> it >> > likely that it can be fixed? It's really killing our development cycle. >> > >> > Bill. >> >>
Re: [julia-users] Test code taking forever to start
I'll see if it is possible. Currently our code does not work at all with large chunks of the Julia commits in that interval. We had to work around various things and don't know precisely when they were switched on or off. Bill. On Wednesday, 23 March 2016 14:54:22 UTC+1, Tim Holy wrote: > > If you can git-bisect the change, it would be a huge help. > > Best, > --Tim > > On Wednesday, March 23, 2016 06:18:23 AM 'Bill Hart' via julia-users > wrote: > > In very recent Julia-0.5-dev the test code in our Nemo module takes > forever > > to start running. It's close to 2 minutes. > > > > This compares with about 15s with older Julia-0.5-dev, say 3 months ago > > before the LLVM switchover. > > > > Does anyone know why there is this massive performance regression. Is it > > likely that it can be fixed? It's really killing our development cycle. > > > > Bill. > >
Re: [julia-users] Test code taking forever to start
If you can git-bisect the change, it would be a huge help. Best, --Tim On Wednesday, March 23, 2016 06:18:23 AM 'Bill Hart' via julia-users wrote: > In very recent Julia-0.5-dev the test code in our Nemo module takes forever > to start running. It's close to 2 minutes. > > This compares with about 15s with older Julia-0.5-dev, say 3 months ago > before the LLVM switchover. > > Does anyone know why there is this massive performance regression. Is it > likely that it can be fixed? It's really killing our development cycle. > > Bill.
[julia-users] Test code taking forever to start
In very recent Julia-0.5-dev the test code in our Nemo module takes forever to start running. It's close to 2 minutes. This compares with about 15s with older Julia-0.5-dev, say 3 months ago before the LLVM switchover. Does anyone know why there is this massive performance regression. Is it likely that it can be fixed? It's really killing our development cycle. Bill.