Re: [MTT users] nightly OMPI tarballs
Thanks Tim! On Nov 21, 2006, at 2:40 PM, Tim Mattox wrote: Hello All, I have moved the nightly tarball generation to start at 9pm US eastern time. At current tarball creation times, it is safe to grab fresh tarballs by 10:30pm assuming we have no more than three (trunk, v1.1, v1.2) that are queued for generation. If you want to live on the edge, they currently take about 23-25 minutes each to make... Happy thanksgiving! P.S. - If anyone would care to investigate a way to improve the tar build time, please take a look at the contrib/nightly/create_tarball.sh script which is in the trunk's SVN that is used to perform this nightly ritual. The bulk of the time is spent in these three commands: autogen.sh configure make -j 4 distcheck But there is some inefficiency afterwards in the generation of the md5sum & sha1sum, since it recomputes them for each tarfile found in the destination directory (the web dowload page), not just for the new tarfile. On 11/8/06, Jeff Squyres wrote: How about the following: 1. Move to earlier, such as 9pm US eastern (want to see what west coasters think of this, but I'm betting they don't care). This can be done more-or-less immediately. 2. We implement MTT ticket #126 (will require some help from the server-side tarball generation, but will be pretty easy). This will definitely need to wait until post-SC. On Nov 8, 2006, at 12:34 PM, Tim Mattox wrote: It would help us here at IU for MTT as well if the tarball generation was a little earlier each day. 9pm Indiana/Eastern time would be good I think. That would make it 6pm West coast time... Does that work for the West coasters? or should we do 10pm Eastern/7pm West? Gleb, George, hpcstork, as three that I know do svn commits outside the typical US workday, how would this affect you? Maybe we could make the non-trunk tarballs even earlier, since the gatekeepers would know when we were "done for the day". What time would Sun need to have the 1.2 tarballs ready for them to do their MTT runs? 7pm Eastern? I can work on making the tarball generation go more quickly, but I suspect I can't get it reliably faster than 1 hour, especially if we have changes on all three branches (trunk, v1.1, v1.2). I have some ideas on how to speed it up though from it's current 2 hour span. One of the ideas, is to have the v1.2 (and maybe v1.1) tarballs be built earlier, so that we only have one tarball to build at the designated time. As for doing multiple builds per day, I am a bit apposed to doing that on a regular basis, for two reasons: 1) It takes time & resources (both human and computer) per tarball for testing, and to look at the results from the testing. One set per day seems at the moment to be what we as a group can currently handle. 2) If we have different groups testing from different tarball sets, then it would become harder to aggregate the testing results, since we would not necessarily be testing the same tarball. On 11/8/06, Jeff Squyres wrote: I'm wondering if it's worthwhile to either a) move back the nightly tarball generation to, say, 9pm US Indiana time or b) perhaps make the tarballs at multiple times during the day. Since we're doing more and more testing, it seems like we need more time to do it before the 9am reports. Right now, we're pretty limited to starting at about 2am (to guarantee that the tarballs have finished building). If you start before then, you could be testing a tarball that's about a day old. This was happening to sun, for example, who (I just found out) starts their testing at 7pm because they have limited time and access to resources (starting at 7pm lets them finish all their testing by 9am). So what do people think about my proposals from above? Either 9pm, or perhaps make them every 6 hours throughout the day. -- Jeff Squyres Server Virtualization Business Unit Cisco Systems ___ mtt-users mailing list mtt-us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users -- Tim Mattox - http://homepage.mac.com/tmattox/ tmat...@gmail.com || timat...@open-mpi.org I'm a bright... http://www.the-brights.net/ ___ mtt-users mailing list mtt-us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users -- Jeff Squyres Server Virtualization Business Unit Cisco Systems ___ mtt-users mailing list mtt-us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users -- Tim Mattox - http://homepage.mac.com/tmattox/ tmat...@gmail.com || timat...@open-mpi.org I'm a bright... http://www.the-brights.net/ ___ mtt-users mailing list mtt-us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users -- Jeff Squyres Server Virtualization Business Unit Cisco Systems
Re: [MTT users] nightly OMPI tarballs
Hello All, I have moved the nightly tarball generation to start at 9pm US eastern time. At current tarball creation times, it is safe to grab fresh tarballs by 10:30pm assuming we have no more than three (trunk, v1.1, v1.2) that are queued for generation. If you want to live on the edge, they currently take about 23-25 minutes each to make... Happy thanksgiving! P.S. - If anyone would care to investigate a way to improve the tar build time, please take a look at the contrib/nightly/create_tarball.sh script which is in the trunk's SVN that is used to perform this nightly ritual. The bulk of the time is spent in these three commands: autogen.sh configure make -j 4 distcheck But there is some inefficiency afterwards in the generation of the md5sum & sha1sum, since it recomputes them for each tarfile found in the destination directory (the web dowload page), not just for the new tarfile. On 11/8/06, Jeff Squyres wrote: How about the following: 1. Move to earlier, such as 9pm US eastern (want to see what west coasters think of this, but I'm betting they don't care). This can be done more-or-less immediately. 2. We implement MTT ticket #126 (will require some help from the server-side tarball generation, but will be pretty easy). This will definitely need to wait until post-SC. On Nov 8, 2006, at 12:34 PM, Tim Mattox wrote: > It would help us here at IU for MTT as well if the tarball > generation was a > little earlier each day. 9pm Indiana/Eastern time would be good I > think. > That would make it 6pm West coast time... Does that work for > the West coasters? or should we do 10pm Eastern/7pm West? > Gleb, George, hpcstork, as three that I know do svn commits outside > the typical > US workday, how would this affect you? > > Maybe we could make the non-trunk tarballs even earlier, since > the gatekeepers would know when we were "done for the day". > What time would Sun need to have the 1.2 tarballs ready > for them to do their MTT runs? 7pm Eastern? > > I can work on making the tarball generation go more quickly, > but I suspect I can't get it reliably faster than 1 hour, > especially if > we have changes on all three branches (trunk, v1.1, v1.2). > I have some ideas on how to speed it up though from it's > current 2 hour span. One of the ideas, is to have the v1.2 (and > maybe v1.1) > tarballs be built earlier, so that we only have one tarball to build > at the designated time. > > As for doing multiple builds per day, I am a bit apposed to doing that > on a regular basis, for two reasons: > 1) It takes time & resources (both human and computer) per tarball > for testing, and to look at the results from the testing. One set per > day seems at the moment to be what we as a group can currently handle. > 2) If we have different groups testing from different tarball sets, > then it would become harder to aggregate the testing results, > since we would not necessarily be testing the same tarball. > > On 11/8/06, Jeff Squyres wrote: >> I'm wondering if it's worthwhile to either a) move back the nightly >> tarball generation to, say, 9pm US Indiana time or b) perhaps make >> the tarballs at multiple times during the day. >> >> Since we're doing more and more testing, it seems like we need more >> time to do it before the 9am reports. Right now, we're pretty >> limited to starting at about 2am (to guarantee that the tarballs have >> finished building). If you start before then, you could be testing a >> tarball that's about a day old. >> >> This was happening to sun, for example, who (I just found out) starts >> their testing at 7pm because they have limited time and access to >> resources (starting at 7pm lets them finish all their testing by >> 9am). >> >> So what do people think about my proposals from above? Either 9pm, >> or perhaps make them every 6 hours throughout the day. >> >> -- >> Jeff Squyres >> Server Virtualization Business Unit >> Cisco Systems >> >> ___ >> mtt-users mailing list >> mtt-us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users >> > > > -- > Tim Mattox - http://homepage.mac.com/tmattox/ > tmat...@gmail.com || timat...@open-mpi.org > I'm a bright... http://www.the-brights.net/ > ___ > mtt-users mailing list > mtt-us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users -- Jeff Squyres Server Virtualization Business Unit Cisco Systems ___ mtt-users mailing list mtt-us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users -- Tim Mattox - http://homepage.mac.com/tmattox/ tmat...@gmail.com || timat...@open-mpi.org I'm a bright... http://www.the-brights.net/
Re: [MTT users] nightly OMPI tarballs
How about the following: 1. Move to earlier, such as 9pm US eastern (want to see what west coasters think of this, but I'm betting they don't care). This can be done more-or-less immediately. 2. We implement MTT ticket #126 (will require some help from the server-side tarball generation, but will be pretty easy). This will definitely need to wait until post-SC. On Nov 8, 2006, at 12:34 PM, Tim Mattox wrote: It would help us here at IU for MTT as well if the tarball generation was a little earlier each day. 9pm Indiana/Eastern time would be good I think. That would make it 6pm West coast time... Does that work for the West coasters? or should we do 10pm Eastern/7pm West? Gleb, George, hpcstork, as three that I know do svn commits outside the typical US workday, how would this affect you? Maybe we could make the non-trunk tarballs even earlier, since the gatekeepers would know when we were "done for the day". What time would Sun need to have the 1.2 tarballs ready for them to do their MTT runs? 7pm Eastern? I can work on making the tarball generation go more quickly, but I suspect I can't get it reliably faster than 1 hour, especially if we have changes on all three branches (trunk, v1.1, v1.2). I have some ideas on how to speed it up though from it's current 2 hour span. One of the ideas, is to have the v1.2 (and maybe v1.1) tarballs be built earlier, so that we only have one tarball to build at the designated time. As for doing multiple builds per day, I am a bit apposed to doing that on a regular basis, for two reasons: 1) It takes time & resources (both human and computer) per tarball for testing, and to look at the results from the testing. One set per day seems at the moment to be what we as a group can currently handle. 2) If we have different groups testing from different tarball sets, then it would become harder to aggregate the testing results, since we would not necessarily be testing the same tarball. On 11/8/06, Jeff Squyres wrote: I'm wondering if it's worthwhile to either a) move back the nightly tarball generation to, say, 9pm US Indiana time or b) perhaps make the tarballs at multiple times during the day. Since we're doing more and more testing, it seems like we need more time to do it before the 9am reports. Right now, we're pretty limited to starting at about 2am (to guarantee that the tarballs have finished building). If you start before then, you could be testing a tarball that's about a day old. This was happening to sun, for example, who (I just found out) starts their testing at 7pm because they have limited time and access to resources (starting at 7pm lets them finish all their testing by 9am). So what do people think about my proposals from above? Either 9pm, or perhaps make them every 6 hours throughout the day. -- Jeff Squyres Server Virtualization Business Unit Cisco Systems ___ mtt-users mailing list mtt-us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users -- Tim Mattox - http://homepage.mac.com/tmattox/ tmat...@gmail.com || timat...@open-mpi.org I'm a bright... http://www.the-brights.net/ ___ mtt-users mailing list mtt-us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users -- Jeff Squyres Server Virtualization Business Unit Cisco Systems
Re: [MTT users] nightly OMPI tarballs
On Wed, 8 Nov 2006, Jeff Squyres wrote: I'm wondering if it's worthwhile to either a) move back the nightly tarball generation to, say, 9pm US Indiana time or b) perhaps make the tarballs at multiple times during the day. Since we're doing more and more testing, it seems like we need more time to do it before the 9am reports. Right now, we're pretty limited to starting at about 2am (to guarantee that the tarballs have finished building). If you start before then, you could be testing a tarball that's about a day old. This was happening to sun, for example, who (I just found out) starts their testing at 7pm because they have limited time and access to resources (starting at 7pm lets them finish all their testing by 9am). So what do people think about my proposals from above? Either 9pm, or perhaps make them every 6 hours throughout the day. Due to resource constraints from IU, I think building them every 6 hours is a bad idea (it's not cheap to build those tarballs). I think moving to 9pm EDT makes sense. Brian -- Brian Barrett Graduate Student, Open Systems Lab, Indiana University http://www.osl.iu.edu/~brbarret/
Re: [MTT users] nightly OMPI tarballs
It would help us here at IU for MTT as well if the tarball generation was a little earlier each day. 9pm Indiana/Eastern time would be good I think. That would make it 6pm West coast time... Does that work for the West coasters? or should we do 10pm Eastern/7pm West? Gleb, George, hpcstork, as three that I know do svn commits outside the typical US workday, how would this affect you? Maybe we could make the non-trunk tarballs even earlier, since the gatekeepers would know when we were "done for the day". What time would Sun need to have the 1.2 tarballs ready for them to do their MTT runs? 7pm Eastern? I can work on making the tarball generation go more quickly, but I suspect I can't get it reliably faster than 1 hour, especially if we have changes on all three branches (trunk, v1.1, v1.2). I have some ideas on how to speed it up though from it's current 2 hour span. One of the ideas, is to have the v1.2 (and maybe v1.1) tarballs be built earlier, so that we only have one tarball to build at the designated time. As for doing multiple builds per day, I am a bit apposed to doing that on a regular basis, for two reasons: 1) It takes time & resources (both human and computer) per tarball for testing, and to look at the results from the testing. One set per day seems at the moment to be what we as a group can currently handle. 2) If we have different groups testing from different tarball sets, then it would become harder to aggregate the testing results, since we would not necessarily be testing the same tarball. On 11/8/06, Jeff Squyres wrote: I'm wondering if it's worthwhile to either a) move back the nightly tarball generation to, say, 9pm US Indiana time or b) perhaps make the tarballs at multiple times during the day. Since we're doing more and more testing, it seems like we need more time to do it before the 9am reports. Right now, we're pretty limited to starting at about 2am (to guarantee that the tarballs have finished building). If you start before then, you could be testing a tarball that's about a day old. This was happening to sun, for example, who (I just found out) starts their testing at 7pm because they have limited time and access to resources (starting at 7pm lets them finish all their testing by 9am). So what do people think about my proposals from above? Either 9pm, or perhaps make them every 6 hours throughout the day. -- Jeff Squyres Server Virtualization Business Unit Cisco Systems ___ mtt-users mailing list mtt-us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users -- Tim Mattox - http://homepage.mac.com/tmattox/ tmat...@gmail.com || timat...@open-mpi.org I'm a bright... http://www.the-brights.net/
[MTT users] nightly OMPI tarballs
I'm wondering if it's worthwhile to either a) move back the nightly tarball generation to, say, 9pm US Indiana time or b) perhaps make the tarballs at multiple times during the day. Since we're doing more and more testing, it seems like we need more time to do it before the 9am reports. Right now, we're pretty limited to starting at about 2am (to guarantee that the tarballs have finished building). If you start before then, you could be testing a tarball that's about a day old. This was happening to sun, for example, who (I just found out) starts their testing at 7pm because they have limited time and access to resources (starting at 7pm lets them finish all their testing by 9am). So what do people think about my proposals from above? Either 9pm, or perhaps make them every 6 hours throughout the day. -- Jeff Squyres Server Virtualization Business Unit Cisco Systems
