RE: Build CI Considerations
would the accessibility just be a configuration script? I'm not sure how CI is set up, but I assume that we could set up something that points to our repository for that, thus any committer could update the configuration script. Also, I dug a bit into Hudson (I'm not familiar with it), there is MSBuild support (http://wiki.hudson-ci.org/display/HUDSON/MSBuild+Plugin), and looking at the FAQ for Hudson that the apache infrastructure team has it looks like plugin's can be added (http://wiki.apache.org/general/Hudson#How_do_I_install_a_new_Hudson_plugin.3F). This is likely something they had to do for us, but I don't see why it would be an issue for them to do it for us ~Prescott Date: Sat, 29 Jan 2011 15:49:51 -0500 Subject: Re: Build CI Considerations From: mhern...@o19s.com To: lucene-net-dev@lucene.apache.org Something to think about for future use is accessibility of the build server for those maintaining. How we do go about sharing that information, obviously you don't want to hand out access to the ci server to a public mailing list. On Fri, Jan 28, 2011 at 1:41 PM, Wyatt Barnett wrote: Why do I forget to check against the obvious? Anyhow, I guess we can run with what we have, though that build is not doing much. Any idea how we get administrative access over there? Might as well try and get it to do stuff like run the tests as well. I think long-term, we'll need something a bit more custom, at least for the build slave/build agent angle --- that would be where the magic largely happens. Such as automated sharpen builds . . . On Thu, Jan 27, 2011 at 5:29 AM, digy digy wrote: No. It's Rune's work. http://mail-archives.apache.org/mod_mbox/lucene-lucene-net-dev/200912.mbox/%3c4b1820f4.10...@gmail.com%3E http://mail-archives.apache.org/mod_mbox/lucene-lucene-net-dev/200912.mbox/%3c4b1820f4.10...@gmail.com%3E DIGY On Thu, Jan 27, 2011 at 12:16 PM, Glyn Darkin wrote: The guys at code better run a Team City CI which has been building Lucene.Net for a while I believe that DIGY set this up. http://teamcity.codebetter.com/login.html Glyn On 27 Jan 2011, at 09:28, Stefan Bodewig wrote: On 2011-01-26, Wyatt Barnett wrote: 2) CI : oh hells yeah. My vision would be to setup something where the automated conversion would be triggered by commits to the stable branch of the java project. I think if we can construct this bit right we can even really get down the road of automatically running all the conversion options until we get it right. Sounds good. Back to the mundane as you said later the ASF runs a few options for CI , one of them is Hudson which has at least one Windows slave installation (Server 2008) and is supposed to support MSBuild. Buildbot might work as well. I'm not up to speed with the state of xbuild but adding support for it to Gump (which fills quite a different role from a traditional CI) wouldn't be too hard and give us builds on Mono - albeit 2.4 right now, this could be changed by adding the Mono PPAs to the Ubuntu servers. Stefan Glyn Darkin Darkin Systems Ltd Mob: 07961815649 Fax: 08717145065 Web: www.darkinsystems.com Company No: 6173001 VAT No: 906350835 -- Michael Herndon Senior Developer (mhern...@o19s.com) 804.767.0083 [connect online] http://www.opensourceconnections.com http://www.amptools.net http://www.linkedin.com/pub/michael-herndon/4/893/23 http://www.facebook.com/amptools.net http://www.twitter.com/amptools-net
Re: Build CI Considerations
On 2011-01-26, Wyatt Barnett wrote: 2) CI : oh hells yeah. My vision would be to setup something where the automated conversion would be triggered by commits to the stable branch of the java project. I think if we can construct this bit right we can even really get down the road of automatically running all the conversion options until we get it right. Sounds good. Back to the mundane as you said later the ASF runs a few options for CI http://ci.apache.org/, one of them is Hudson https://hudson.apache.org/hudson/ which has at least one Windows slave installation (Server 2008) and is supposed to support MSBuild. Buildbot might work as well. I'm not up to speed with the state of xbuild but adding support for it to Gump (which fills quite a different role from a traditional CI) wouldn't be too hard and give us builds on Mono - albeit 2.4 right now, this could be changed by adding the Mono PPAs to the Ubuntu servers. Stefan
Re: Build CI Considerations
The guys at code better run a Team City CI which has been building Lucene.Net for a while I believe that DIGY set this up. http://teamcity.codebetter.com/login.html Glyn On 27 Jan 2011, at 09:28, Stefan Bodewig wrote: On 2011-01-26, Wyatt Barnett wrote: 2) CI : oh hells yeah. My vision would be to setup something where the automated conversion would be triggered by commits to the stable branch of the java project. I think if we can construct this bit right we can even really get down the road of automatically running all the conversion options until we get it right. Sounds good. Back to the mundane as you said later the ASF runs a few options for CI http://ci.apache.org/, one of them is Hudson https://hudson.apache.org/hudson/ which has at least one Windows slave installation (Server 2008) and is supposed to support MSBuild. Buildbot might work as well. I'm not up to speed with the state of xbuild but adding support for it to Gump (which fills quite a different role from a traditional CI) wouldn't be too hard and give us builds on Mono - albeit 2.4 right now, this could be changed by adding the Mono PPAs to the Ubuntu servers. Stefan Glyn Darkin Darkin Systems Ltd Mob: 07961815649 Fax: 08717145065 Web: www.darkinsystems.com Company No: 6173001 VAT No: 906350835
Re: Build CI Considerations
No. It's Rune's work. http://mail-archives.apache.org/mod_mbox/lucene-lucene-net-dev/200912.mbox/%3c4b1820f4.10...@gmail.com%3E http://mail-archives.apache.org/mod_mbox/lucene-lucene-net-dev/200912.mbox/%3c4b1820f4.10...@gmail.com%3E DIGY On Thu, Jan 27, 2011 at 12:16 PM, Glyn Darkin g...@darkinsystems.comwrote: The guys at code better run a Team City CI which has been building Lucene.Net for a while I believe that DIGY set this up. http://teamcity.codebetter.com/login.html Glyn On 27 Jan 2011, at 09:28, Stefan Bodewig wrote: On 2011-01-26, Wyatt Barnett wrote: 2) CI : oh hells yeah. My vision would be to setup something where the automated conversion would be triggered by commits to the stable branch of the java project. I think if we can construct this bit right we can even really get down the road of automatically running all the conversion options until we get it right. Sounds good. Back to the mundane as you said later the ASF runs a few options for CI http://ci.apache.org/, one of them is Hudson https://hudson.apache.org/hudson/ which has at least one Windows slave installation (Server 2008) and is supposed to support MSBuild. Buildbot might work as well. I'm not up to speed with the state of xbuild but adding support for it to Gump (which fills quite a different role from a traditional CI) wouldn't be too hard and give us builds on Mono - albeit 2.4 right now, this could be changed by adding the Mono PPAs to the Ubuntu servers. Stefan Glyn Darkin Darkin Systems Ltd Mob: 07961815649 Fax: 08717145065 Web: www.darkinsystems.com Company No: 6173001 VAT No: 906350835
Re: Build CI Considerations
Robert, . I don't believe this is necessary. At least there were no requests for alternative build systems in the past. There may never be a need for the alternative building scripts, its was more of a curious question. I've seen a few projects on github use both albacore and psake. Maybe it was done in vain by the authors or possibly to attract people from the alt.net crowd. The Mono ecosystem is also preferring MSBuild/Visual Studio projects to other build systems. Are they using the xbuild or literally using msbuild and are they completely compatible ? I haven't kept up with how far xbuild has come lately. I know that mono develop and mono is still a little flakey with c# 4.0 and things like the expando object but that probably will not affect the current lucene builds. I believe we should rather provide our own NuGet package than using it for external dependencies. We currently have only one dependency: NUnit. - Robert So I think is in more regards to dependencies in general. This has a possibility to increase, especially if we incorporate other libraries such as Moq for testing, or expand lucene functionality to include linq. Nuget isn't just for compiled assemblies but can also be used for importing classes such as the notorious internal Guard class. http://www.clariusconsulting.net/blogs/kzu/archive/2010/12/08/HowtocreatelightweightreusablesourcecodewithNuGet.aspx Regarding NuGet for dependencies.. I'd say we should not do that at this time. Primarily because, AFAIK, NuGet is an add-on for Visual Studio which can only be used in a paid-for version of 2010. You can't use it with the Express versions. You can't use it in MonoDevelop. So, we'd be raising the barrier of entry to only those who have a licensed version of Visual Studio 2010. - Troy Troy, The core of Nuget itself does not actually have a ui or dependency on Visual Studio. While nuget's default interfaces are currently limited to powershell or visual studio 2010, it is possible to create additional user interfaces for it. Its just a matter of time before there is a command line version of nuget. There has already been patches made to the nuget core to support mono and there already hooks in nuget aimed at command line usage in the future. However, another thing to pay attention to is that once the binaries are added to the project, they become a part of the source tree. Thus people can still build and code without using Nuget. While we do not need to run or rush to this right away, its something to be considered. CI As far as CI is concerned, I'm not particular on any technology. Hudson or cc.net, or insert another one here, is all fine with me. I would just like to see one become a part of the process especially when we are looking at using tools like Sharpen. Build targets: Real question is what do we want to target. I do think we can target different versions for development work versus runtime compatibility. EG, nothing is stopping us from making the dev/build environment VS 2010 while still targeting the 2.0 runtime. Personally, I'd vote aiming for 3.5/VS 2010, at least for the 3.0 branch. - Wyatt Wyatt, I know that its possible to load the same csproj files for both VS 2008 and 2010 (using different solutions). These will generally load inside of Mono Develop as well. VS 2005 is where it gets tricky. Another thing we should look at is making sure can compile on both the 2x and 4x runtime. I remember getting log4net to compile in 4x was a little bit of work. I am sure other would like for this to build on other versions of the framework. I remember CF being one of those mentioned. I don't know if the list includes Silverlight or Micro. If it does we'll most likely need to look hard at the BCL as well as the Supported Interfaces, core types and methods. Possibly the Portable Tools Library could help with that in the future. http://blogs.msdn.com/b/bclteam/archive/2011/01/19/announcing-portable-library-tools-ctp-justin-van-patten.aspx - Michael. Below are the responses from another thread: Proposal Status, Initial Committors List, Contributors List Robert Jordan, Michael, On 26.01.2011 19:18, Michael Herndon wrote: Should the project include build scripts for powershell, nant, albacore scripts? (would this further entice developers to download the code and build it)? These also could be available as a separate download. I don't believe this is necessary. At least there were no requests for alternative build systems in the past. What would be the preferred way of working with the code base in mono ? Should we ensure the project loads and compiles in mono develop, etc. The Mono ecosystem is also preferring MSBuild/Visual Studio projects to other build systems. would it be beneficial to add a .gitignore file for those avid git-svn users? Yes, but let's do it on demand. should we start looking at nuget for external
Build CI Considerations
Per Michael's suggestion I'm branching this off into a new thread. I'll start by saying this is somewhere I think I could help alot -- I'm still a recovering liberal arts major so I can't claim to grok too much of the underlying computer sciency bits. I've also spent as much if not more time in IT managing infrastructure and processes than building software. Finally, I've got some facilities access that could be helpful in a pinch. Anyhow, here goes: 1) Build targets: Real question is what do we want to target. I do think we can target different versions for development work versus runtime compatibility. EG, nothing is stopping us from making the dev/build environment VS 2010 while still targeting the 2.0 runtime. Personally, I'd vote aiming for 3.5/VS 2010, at least for the 3.0 branch. 2) CI : oh hells yeah. My vision would be to setup something where the automated conversion would be triggered by commits to the stable branch of the java project. I think if we can construct this bit right we can even really get down the road of automatically running all the conversion options until we get it right. Another angle to this is that, if it is the paid tool we need, we only need the build agent to have a license -- it could easily just convert the source and check it in where people could check it out and work on it. Or something like that. But back to the mundane -- I've got alot of seat-time with TeamCity and I'd be happy to setup a build server. I've got the facilities to handle this personally. I think to really get it right we'll have to make some significant changes in the project structure which brings me to .. . 3) What should be in SVN: I'm a big fan of everything and the kitchen sink. Its 2011, server space and bandwidth are cheap. Philosophically, checkout - build should be a seamless, painless process on this sort of library-style project presuming basic prerequisites. I like the idea of including the .gitignore (and .hgignore) files. Once setup, they really don't change unless you are really re-working stuff. And given the setup with a restricted SVN master we are going to leave lots of patches in the street if we can't be DCVS friendly. 4) Mono: my mac-filesystem-foo failed me, but I think the bits I uploaded wrt issue https://issues.apache.org/jira/browse/LUCENENET-377 should compile on Mono using the build script included pointed at the appropriate tool. In any case, this can easily be part of a teamcity build process. We can even do it on linux if we'd like. Hope this helps.
Re: Build CI Considerations
On build systems -- I think MSBuild can definitely get you where you'd want to be -- the triple-targeted version I posted had a very rough MSBuild deployment script. The alternative build systems really start to shine in places we don't need the help -- we don't need to do complex packaging, build installers, manipulate database servers or message queues. Said alternative build systems also generally call MSBuild to do the actual building of the project so we need a bit of a MSBuild build script anyhow. NuGet is actually avaliable as the standalone command line tool -- see the downloads page on codeplex [http://nuget.codeplex.com/releases/view/57303]. It is meant for publication side but it is a complete client near as I can tell. In addition, you can still use NuGet to manage packages and other dependencies and not require it be installed on the clients. When you install a nuget package, the bits are pulled locally into the solution, alls we need to do from there is to check said bits into the solution and they can ride with everything else in SVN. Speaking of dependencies, there are actually two -- sharpziplib is required by the tests. This is actually a bit of an issue as it is verboten to ship it with the project so it screws the CI pooch. Or at least the alt.net/agilist one I subscribe to. Is there a requirement to use Apache for CI? Or can we go out on our own? I suspect they both can get us there, but if we are going to be pulling off stunts like incorporating the sharpen (or whatever) port into things we'll probably need a bit more control of the build agent machine than apache can provide. I'm also fairly confident I can get a complex, multi-step, multi-platform and perhaps even multi-build-agent build process working in TeamCity. Has anyone here done such a thing in Hudson? I was having some success with .csproj files working all the way up and down from VS2005 thru VS2010 without any conflicts with this project. It seems that VS2008/2010 adds some stuff to the .csproj file, but VS2005 walks right past that -- my suspicion is that pain had to do with the more complex project types, all we've got is class libraries which haven't changed since the days when men were men and cars were cars. I tried everything I could to break the All that said, is there anyone in this room who needs to *develop* on the solution in VS2005? Or is the requirement can build in a pure 2.0 environment and be used in a pure 2.0 environment? Good point on getting 4.0 into the process as well and this is also something CI can cover easily. My 2c on silverlight and compact framework clients would be we are getting way ahead of ourselves -- we don't have a working conversion process for the full framework yet, not sure if we want to open that can of worms. Moreover, in terms of usage profiles, chances are major search operations you'd want to do from a silverlight or CF app would get pushed onto the cloud server somewhere. WP7 would be a more interesting angle, but until that gets local storage abilities it is kind of pointless to even consider. On Wed, Jan 26, 2011 at 4:56 PM, Michael Herndon mhern...@o19s.com wrote: Robert, . I don't believe this is necessary. At least there were no requests for alternative build systems in the past. There may never be a need for the alternative building scripts, its was more of a curious question. I've seen a few projects on github use both albacore and psake. Maybe it was done in vain by the authors or possibly to attract people from the alt.net crowd. The Mono ecosystem is also preferring MSBuild/Visual Studio projects to other build systems. Are they using the xbuild or literally using msbuild and are they completely compatible ? I haven't kept up with how far xbuild has come lately. I know that mono develop and mono is still a little flakey with c# 4.0 and things like the expando object but that probably will not affect the current lucene builds. I believe we should rather provide our own NuGet package than using it for external dependencies. We currently have only one dependency: NUnit. - Robert So I think is in more regards to dependencies in general. This has a possibility to increase, especially if we incorporate other libraries such as Moq for testing, or expand lucene functionality to include linq. Nuget isn't just for compiled assemblies but can also be used for importing classes such as the notorious internal Guard class. http://www.clariusconsulting.net/blogs/kzu/archive/2010/12/08/HowtocreatelightweightreusablesourcecodewithNuGet.aspx Regarding NuGet for dependencies.. I'd say we should not do that at this time. Primarily because, AFAIK, NuGet is an add-on for Visual Studio which can only be used in a paid-for version of 2010. You can't use it with the Express versions. You can't use it in MonoDevelop. So, we'd be raising the barrier of entry to only those who have a licensed version of Visual Studio 2010. - Troy Troy,
