[oi-dev] Distribution build system
Hi All The project is moving forward very slowly. I my opinion one of the main reasons behind it is lack of unified distribution build system. Previous releases were put together manually. Probably one or two core contributors would be be able to repeat the whole process at this point without investing significant amount of time into learning how things fit together. We need distribution release and publishing process to be automated and repeatable. There is a significant amount of bug reports in OpenIndiana bug tracker. Many of bugs require very simple changes to get fixed. URL updates or branding updates for example. Many contributors are more than capable of fixing such bugs. Because it is not clear what goes where, and also how to build and then test things many contributors simply don't bother looking at bugs at all. I am proposing creating a unified distribution build system. A system that would build the whole distribution after issuing a single "make publish" or similar command. Having such system will let us to release early, release often. It should improve the development progress in general. Bugfixes and security updates would get integrated in no time. New users would have an easy start. We could point new contributors at the build system and simply ask them to start hacking. Having all consolidations referenced from a single build system would make it clear to them where things go. Base system changes, etc. - core consolidations, new software - add-on consolidation directory, and so on. Continuous build system could be implemented using the same tools on the OpenIndiana build machines, including the SPARC ones. Many people will say that this is not possible and that even Oracle is not doing it. I say such system is possible but will require significant amount of work and significant amount of time to prepare. It will be worth the effort if done right. All major open source OS distributions have the release process automated in one way or another. I think this is the key to our success. Splitting the build and release engineering process between consolidation maintainers/owners based on Oracle model proved to not to work well for us. I would like to start a dialog between the core contributors about such build system. Discuss whether it is needed or not. And if the decision is made that it is needed - discuss requirements, technical details and then actually implement it! Andrzej ___ oi-dev mailing list oi-dev@openindiana.org http://openindiana.org/mailman/listinfo/oi-dev
Re: [oi-dev] Distribution build system
Great idea, worth the pain! On Wed, Mar 16, 2011 at 11:35 AM, Andrzej Szeszo wrote: > Hi All > > The project is moving forward very slowly. I my opinion one of the main > reasons behind it is lack of unified distribution build system. > > Previous releases were put together manually. Probably one or two core > contributors would be be able to repeat the whole process at this point > without investing significant amount of time into learning how things fit > together. We need distribution release and publishing process to be > automated and repeatable. > > There is a significant amount of bug reports in OpenIndiana bug tracker. > Many of bugs require very simple changes to get fixed. URL updates or > branding updates for example. Many contributors are more than capable of > fixing such bugs. Because it is not clear what goes where, and also how to > build and then test things many contributors simply don't bother looking at > bugs at all. > > I am proposing creating a unified distribution build system. A system that > would build the whole distribution after issuing a single "make publish" or > similar command. > > Having such system will let us to release early, release often. It should > improve the development progress in general. Bugfixes and security updates > would get integrated in no time. New users would have an easy start. We > could point new contributors at the build system and simply ask them to > start hacking. Having all consolidations referenced from a single build > system would make it clear to them where things go. Base system changes, > etc. - core consolidations, new software - add-on consolidation directory, > and so on. > > Continuous build system could be implemented using the same tools on the > OpenIndiana build machines, including the SPARC ones. > > Many people will say that this is not possible and that even Oracle is not > doing it. I say such system is possible but will require significant amount > of work and significant amount of time to prepare. It will be worth the > effort if done right. All major open source OS distributions have the > release process automated in one way or another. I think this is the key to > our success. Splitting the build and release engineering process between > consolidation maintainers/owners based on Oracle model proved to not to work > well for us. > > I would like to start a dialog between the core contributors about such > build system. Discuss whether it is needed or not. And if the decision is > made that it is needed - discuss requirements, technical details and then > actually implement it! > > Andrzej > > ___ > oi-dev mailing list > oi-dev@openindiana.org > http://openindiana.org/mailman/listinfo/oi-dev > ___ oi-dev mailing list oi-dev@openindiana.org http://openindiana.org/mailman/listinfo/oi-dev
Re: [oi-dev] Distribution build system
A laudable aim but do we have the man power to do it, without negatively affecting our existing schedules for illumos and stable releases? Bye, Deano From: Colin Ellis [mailto:panamaya...@gmail.com] Sent: 16 March 2011 11:46 To: OpenIndiana Developer mailing list Subject: Re: [oi-dev] Distribution build system Great idea, worth the pain! On Wed, Mar 16, 2011 at 11:35 AM, Andrzej Szeszo wrote: Hi All The project is moving forward very slowly. I my opinion one of the main reasons behind it is lack of unified distribution build system. Previous releases were put together manually. Probably one or two core contributors would be be able to repeat the whole process at this point without investing significant amount of time into learning how things fit together. We need distribution release and publishing process to be automated and repeatable. There is a significant amount of bug reports in OpenIndiana bug tracker. Many of bugs require very simple changes to get fixed. URL updates or branding updates for example. Many contributors are more than capable of fixing such bugs. Because it is not clear what goes where, and also how to build and then test things many contributors simply don't bother looking at bugs at all. I am proposing creating a unified distribution build system. A system that would build the whole distribution after issuing a single "make publish" or similar command. Having such system will let us to release early, release often. It should improve the development progress in general. Bugfixes and security updates would get integrated in no time. New users would have an easy start. We could point new contributors at the build system and simply ask them to start hacking. Having all consolidations referenced from a single build system would make it clear to them where things go. Base system changes, etc. - core consolidations, new software - add-on consolidation directory, and so on. Continuous build system could be implemented using the same tools on the OpenIndiana build machines, including the SPARC ones. Many people will say that this is not possible and that even Oracle is not doing it. I say such system is possible but will require significant amount of work and significant amount of time to prepare. It will be worth the effort if done right. All major open source OS distributions have the release process automated in one way or another. I think this is the key to our success. Splitting the build and release engineering process between consolidation maintainers/owners based on Oracle model proved to not to work well for us. I would like to start a dialog between the core contributors about such build system. Discuss whether it is needed or not. And if the decision is made that it is needed - discuss requirements, technical details and then actually implement it! Andrzej ___ oi-dev mailing list oi-dev@openindiana.org http://openindiana.org/mailman/listinfo/oi-dev ___ oi-dev mailing list oi-dev@openindiana.org http://openindiana.org/mailman/listinfo/oi-dev
Re: [oi-dev] Distribution build system
On 16 Mar 2011, at 12:12, Deano wrote: > A laudable aim but do we have the man power to do it, without negatively > affecting our existing schedules for illumos and stable releases? Can we afford *not* to do it? Anything to make it easier and more practical for contributions the better... Chris ___ oi-dev mailing list oi-dev@openindiana.org http://openindiana.org/mailman/listinfo/oi-dev
Re: [oi-dev] Distribution build system
On 03/16/11 13:34, Chris Ridd wrote: On 16 Mar 2011, at 12:12, Deano wrote: > A laudable aim but do we have the man power to do it, without negatively affecting our existing schedules for illumos and stable releases? Can we afford*not* to do it? Anything to make it easier and more practical for contributions the better... We could start from the top, automate distro-importer and distro-constructor first while using pre-compiled components. Then gradually add consolidation build systems to the tree. This is just one of the possible approaches. We definitely need automation/repeatability if we are thinking about maintaining stable release. I would rather have the new release assembled using a new automated system even if it means getting the release out later. Andrzej ___ oi-dev mailing list oi-dev@openindiana.org http://openindiana.org/mailman/listinfo/oi-dev
Re: [oi-dev] Distribution build system
On 16 Mar 2011, at 15:43, Andrzej Szeszo wrote: > On 03/16/11 13:34, Chris Ridd wrote: >> On 16 Mar 2011, at 12:12, Deano wrote: >> >> >>> > >>> A laudable aim but do we have the man power to do it, without negatively >>> affecting our existing schedules for illumos and stable releases? >>> >> Can we afford *not* >> to do it? Anything to make it easier and more practical for contributions >> the better... >> > > We could start from the top, automate distro-importer and distro-constructor > first while using pre-compiled components. Then gradually add consolidation > build systems to the tree. This is just one of the possible approaches. > > We definitely need automation/repeatability if we are thinking about > maintaining stable release. I completely agree. FWIW at work we have an XML file describing each release as (essentially) a number of git repos + tags to checkout, and then what script to run to build each given repo. Our overall build script goes through that XML in order, and is able to do "the right thing". This gives us pretty reproducible releases. Chris ___ oi-dev mailing list oi-dev@openindiana.org http://openindiana.org/mailman/listinfo/oi-dev
Re: [oi-dev] Distribution build system
On 03/16/11 09:12 AM, Chris Ridd wrote: > FWIW at work we have an XML file describing each release as (essentially) a > number of git repos + tags to checkout, and then what script to run to build > each given repo. Our overall build script goes through that XML in order, and > is able to do "the right thing". This gives us pretty reproducible releases. Sounds very much like jhbuild, which is often used to build projects like GNOME & X.Org with dozens or hundreds of individual modules to checkout and build. http://www.freedesktop.org/wiki/Software/jhbuild -- -Alan Coopersmith-alan.coopersm...@oracle.com Oracle Solaris Platform Engineering: X Window System ___ oi-dev mailing list oi-dev@openindiana.org http://openindiana.org/mailman/listinfo/oi-dev
Re: [oi-dev] Distribution build system
Thanks for the link Alan. We will definitely evaluate it. Andrzej On Wednesday, 16 March 2011, Alan Coopersmith wrote: > On 03/16/11 09:12 AM, Chris Ridd wrote: >> FWIW at work we have an XML file describing each release as (essentially) a >> number of git repos + tags to checkout, and then what script to run to build >> each given repo. Our overall build script goes through that XML in order, >> and is able to do "the right thing". This gives us pretty reproducible >> releases. > > Sounds very much like jhbuild, which is often used to build projects > like GNOME & X.Org with dozens or hundreds of individual modules to > checkout and build. > > http://www.freedesktop.org/wiki/Software/jhbuild > > -- > -Alan Coopersmith- alan.coopersm...@oracle.com > Oracle Solaris Platform Engineering: X Window System > > > ___ > oi-dev mailing list > oi-dev@openindiana.org > http://openindiana.org/mailman/listinfo/oi-dev > ___ oi-dev mailing list oi-dev@openindiana.org http://openindiana.org/mailman/listinfo/oi-dev
Re: [oi-dev] Distribution build system
On 16 Mar 2011, at 16:34, Alan Coopersmith wrote: > On 03/16/11 09:12 AM, Chris Ridd wrote: >> FWIW at work we have an XML file describing each release as (essentially) a >> number of git repos + tags to checkout, and then what script to run to build >> each given repo. Our overall build script goes through that XML in order, >> and is able to do "the right thing". This gives us pretty reproducible >> releases. > > Sounds very much like jhbuild, which is often used to build projects > like GNOME & X.Org with dozens or hundreds of individual modules to > checkout and build. > > http://www.freedesktop.org/wiki/Software/jhbuild Yes, that seems somewhat similar. jhbuild looks like it has some builtin support for running as a buildbot slave, so if we also want a continuous build system then it could be useful for that too. The only downside I can see is that the jhbuild configuration file format is not very formally structured (eg XML). That means it is harder to do a few things like mechanically generate new release files based on another file (eg create a snapshot release based on the stable branch), or compare different release files, but that may not be a big problem. Thanks for the link! Chris ___ oi-dev mailing list oi-dev@openindiana.org http://openindiana.org/mailman/listinfo/oi-dev
Re: [oi-dev] Distribution build system
On 03/16/11 11:35 AM, Andrzej Szeszo wrote: Hi All The project is moving forward very slowly. I my opinion one of the main reasons behind it is lack of unified distribution build system. I absolutely agree - right now it's far too hard for people to build the OS; not only does this hinder new developers getting involved, it hinders everybody. Previous releases were put together manually. Probably one or two core contributors would be be able to repeat the whole process at this point without investing significant amount of time into learning how things fit together. We need distribution release and publishing process to be automated and repeatable. Absolutely agree. There is a significant amount of bug reports in OpenIndiana bug tracker. Many of bugs require very simple changes to get fixed. URL updates or branding updates for example. Many contributors are more than capable of fixing such bugs. Because it is not clear what goes where, and also how to build and then test things many contributors simply don't bother looking at bugs at all. I am proposing creating a unified distribution build system. A system that would build the whole distribution after issuing a single "make publish" or similar command. This is exactly what we need. Having such system will let us to release early, release often. It should improve the development progress in general. Bugfixes and security updates would get integrated in no time. New users would have an easy start. We could point new contributors at the build system and simply ask them to start hacking. Having all consolidations referenced from a single build system would make it clear to them where things go. Base system changes, etc. - core consolidations, new software - add-on consolidation directory, and so on. *nods* That way we can have a standard for how things are laid out on the filesystem, for how patches are applied, what environment is used, etc. This will also ensure everyone is building things in the same way, which will reduce time wasted diagnosing build issues. Continuous build system could be implemented using the same tools on the OpenIndiana build machines, including the SPARC ones. Continuous integration would be an enormous win for the project - it would mean that as soon as Oracle commit some code that breaks our patch set, the responsible people can be notified and immediately work on the issue. This would ensure that when we come to build a new /dev release, that there is a lot less work to do. It would also let us spread the work out more easily. Many people will say that this is not possible and that even Oracle is not doing it. I say such system is possible but will require significant amount of work and significant amount of time to prepare. It will be worth the effort if done right. All major open source OS distributions have the release process automated in one way or another. I think this is the key to our success. Splitting the build and release engineering process between consolidation maintainers/owners based on Oracle model proved to not to work well for us. Anything is possible when it comes to code, and this is no exception. Anyone who says it can't or shouldn't be done is wrong. If Oracle aren't doing it, its because they are behind the times. We should work "smarter, not harder". Working smarter is definitely the key to our success. We are "stuck in the mud" until we do this. I would like to start a dialog between the core contributors about such build system. Discuss whether it is needed or not. And if the decision is made that it is needed - discuss requirements, technical details and then actually implement it! It is needed. The decision is a no brainer, so it gets my vote. The question is how we implement it. Would anyone be interested in a Hackathon to get this work completed over a weekend in the near future? Alasdair. ___ oi-dev mailing list oi-dev@openindiana.org http://openindiana.org/mailman/listinfo/oi-dev
Re: [oi-dev] Distribution build system
I figure creating a complete build build systems around the build systems of all the different consolidations would be a huge task. I think the best starting point would be to better automate and simplify the distro-construction step, in particular to simplyify adding updates to an existing repo (both distro-importing svr4 packages and merging with other repos) and constructing new live images. -- Guido Berhoerster ___ oi-dev mailing list oi-dev@openindiana.org http://openindiana.org/mailman/listinfo/oi-dev
Re: [oi-dev] Distribution build system
Likely controversial view inline below... -Original Message- From: Alasdair Lumsden [mailto:alasdai...@gmail.com] Sent: 17 March 2011 10:59 To: oi-dev@openindiana.org Subject: Re: [oi-dev] Distribution build system >> Having such system will let us to release early, release often. It >> should improve the development progress in general. Bugfixes and >> security updates would get integrated in no time. New users would have >> an easy start. We could point new contributors at the build system and >> simply ask them to start hacking. Having all consolidations referenced >> from a single build system would make it clear to them where things go. >> Base system changes, etc. - core consolidations, new software - add-on >> consolidation directory, and so on. > >*nods* > >That way we can have a standard for how things are laid out on the >filesystem, for how patches are applied, what environment is used, etc. > >This will also ensure everyone is building things in the same way, which >will reduce time wasted diagnosing build issues. If we are going to this effort than we should IMHO do it probably and break the direct link between OI and consolidations. I can hear the sharp intakes of breath from here, from OI point of view consolidations are just groupings of certain packages and files put together by an upstream provider, they don't necessarily make any sense in term of useable packages down at the users end. Example abound of odd inclusions that made sense to the original consolidation author but not to the user (gruff is an known example). >From a users, developers and installers point of view, I want to get/update/remove X without knowing or caring about consolidations, therefore they should become hidden behind our one stop build system. If we whilst doing this work, we add a new OI thing (deliberately not giving it a name) that provides OI useful chunks and mapping internally through upstream consolidations and builds and grabs the bits of the version it wants, we achieve this independence at the same time. Effectively treating consolidation packages now as just another form of source tree, to mine and use as we see fit. Why go to this effect? Because at the moment we can't control where things go, what versions we use and even what our users can install/remove. By having a single build system work in chunks of OI decided work, we can easily begin to control where/what/how. You press the build button and it knows about upstream consolidation and OI chunks, out pops a distro with exactly in the state we set up, rather than constant fighting upstream changes when and if they change. We take pieces as and when we require them and they meet our demands. In some ways I'm not suggesting dropping consolidations but instead have a system that mine's other consolidations. Adding a 'use file X in consolidation Y' rather than 'depend on consolidation Y' operator. >> Many people will say that this is not possible and that even Oracle is >> not doing it. I say such system is possible but will require significant >> amount of work and significant amount of time to prepare. It will be >> worth the effort if done right. All major open source OS distributions >> have the release process automated in one way or another. I think this >> is the key to our success. Splitting the build and release engineering >> process between consolidation maintainers/owners based on Oracle model >> proved to not to work well for us. > >Anything is possible when it comes to code, and this is no exception. >Anyone who says it can't or shouldn't be done is wrong. > >If Oracle aren't doing it, its because they are behind the times. We >should work "smarter, not harder". Working smarter is definitely the key >to our success. > >We are "stuck in the mud" until we do this. Yep I agree, I think however instead of digging ourselves out of the mud piece by piece, we should look at the tires themselves. >> I would like to start a dialog between the core contributors about such >> build system. Discuss whether it is needed or not. And if the decision >> is made that it is needed - discuss requirements, technical details and >> then actually implement it! > >It is needed. The decision is a no brainer, so it gets my vote. The >question is how we implement it. > >Would anyone be interested in a Hackathon to get this work completed >over a weekend in the near future? If the hackathon is about a month away I'd happily get involved (too busy right now). As a one stop build system is pretty big thing, I do think we should produce a discussion document, with both short and long term goals. I'd rather us spend longer and do it right, then quickly add anot
Re: [oi-dev] Distribution build system
On 17/03/2011 13:26, Deano wrote: If we are going to this effort than we should IMHO do it probably and break the direct link between OI and consolidations. I can hear the sharp intakes of breath from here, from OI point of view consolidations are just groupings of certain packages and files put together by an upstream provider, they don't necessarily make any sense in term of useable packages down at the users end. Example abound of odd inclusions that made sense to the original consolidation author but not to the user (gruff is an known example). From a users, developers and installers point of view, I want to get/update/remove X without knowing or caring about consolidations, therefore they should become hidden behind our one stop build system. If we whilst doing this work, we add a new OI thing (deliberately not giving it a name) that provides OI useful chunks and mapping internally through upstream consolidations and builds and grabs the bits of the version it wants, we achieve this independence at the same time. Effectively treating consolidation packages now as just another form of source tree, to mine and use as we see fit. Why go to this effect? Because at the moment we can't control where things go, what versions we use and even what our users can install/remove. By having a single build system work in chunks of OI decided work, we can easily begin to control where/what/how. You press the build button and it knows about upstream consolidation and OI chunks, out pops a distro with exactly in the state we set up, rather than constant fighting upstream changes when and if they change. We take pieces as and when we require them and they meet our demands. In some ways I'm not suggesting dropping consolidations but instead have a system that mine's other consolidations. Adding a 'use file X in consolidation Y' rather than 'depend on consolidation Y' operator. Breaking the link with upstream consolidations may be the thing we will ultimately have to do. For now, while the development of the consolidations like sfw, jds, xnv, ips, caiman happens in the open we probably want to continue using them in their current form for as long as possible. We don't have manpower to re-create all build recipes and packages at the moment. Creating a wrapper on top of existing consolidations and their build systems would be the simplest thing to do right now. >> Many people will say that this is not possible and that even Oracle is >> not doing it. I say such system is possible but will require significant >> amount of work and significant amount of time to prepare. It will be >> worth the effort if done right. All major open source OS distributions >> have the release process automated in one way or another. I think this >> is the key to our success. Splitting the build and release engineering >> process between consolidation maintainers/owners based on Oracle model >> proved to not to work well for us. > >Anything is possible when it comes to code, and this is no exception. >Anyone who says it can't or shouldn't be done is wrong. > >If Oracle aren't doing it, its because they are behind the times. We >should work "smarter, not harder". Working smarter is definitely the key >to our success. > >We are "stuck in the mud" until we do this. Yep I agree, I think however instead of digging ourselves out of the mud piece by piece, we should look at the tires themselves. >> I would like to start a dialog between the core contributors about such >> build system. Discuss whether it is needed or not. And if the decision >> is made that it is needed - discuss requirements, technical details and >> then actually implement it! > >It is needed. The decision is a no brainer, so it gets my vote. The >question is how we implement it. > >Would anyone be interested in a Hackathon to get this work completed >over a weekend in the near future? If the hackathon is about a month away I'd happily get involved (too busy right now). As a one stop build system is pretty big thing, I do think we should produce a discussion document, with both short and long term goals. I'd rather us spend longer and do it right, then quickly add another complicated layer that makes sense now but will just become another burden down the line. I reckon we quickly figure out short list of requirements and start implementing the system. From top to bottom. Starting with the distro-importer and distro-constructor, just like Guido is suggesting in another message. Then move on to adding support for the actual build systems. I wasn't involved during the OSol era so I can't comment on whether the current overall distribution design was useful then, but at this moment, I'm not sure I see its strengths. Its seems instead of taking the amazing source and tech we have and pushing it forward we are largely fighting to keep our head above water and I largely see that as a fault of the development and distribution model which doesn't
Re: [oi-dev] Distribution build system
Hello, The conversation on the Distributed build system stalled on finer points but the large idea seemed to have everyone support. A continuous build system, that automatically went through all consolidations would be a major step forward for the OI development process. So lets get things moved on a least for the most basic system we all agree would be useful. I looked into some more advanced build systems (Scons, WAF, etc.) but tbh all were slightly overkill for what the basic simple system needs which is simply a makefile or script that grabs each consolidation, sets up the environment, builds and moves onto the next (with appropriate error handling etc.) Aszezo has some idea and how to get this party started I believe, so probably best for him to make more concrete suggestions? If we could get a empty framework in place and running, we could then pop in each consolidation in turn when ready, and therefore gradually get our continuous build system working, the particularly troublesome ones don't have to stop the easier ones. A hack-a-thon might make an ideal time for people to get together and push this all together? Thoughts please, Deano ___ oi-dev mailing list oi-dev@openindiana.org http://openindiana.org/mailman/listinfo/oi-dev
Re: [oi-dev] Distribution build system
Hi, Deano I am converting all the consolidations building instruction into xml files. and this xml file can be playbacked but /opt/TWWfsw/bin/sb (sb=software build) tool. Very similar following command to create O.I. CD/USB image. except the abstraction is one level higher. pfexec distro_const build ./new_slim_cd_x86.xml So for this digitization effort is to generate O.I. sparc image to provide alternative OS that our Sparc hardware can run freely. I don't expect it got adopted by this group since TWW Inc.'s GNU tool-sets is not well-known. But I am documenting this approach at R1. tj R1: http://wiki.openindiana.org/oi/CPAMTWW On Wed, Apr 6, 2011 at 7:13 AM, Deano wrote: > Hello, > > The conversation on the Distributed build system stalled on finer points but > the large idea seemed to have everyone support. A continuous build system, > that automatically went through all consolidations would be a major step > forward for the OI development process. > So lets get things moved on a least for the most basic system we all agree > would be useful. I looked into some more advanced build systems (Scons, WAF, > etc.) but tbh all were slightly overkill for what the basic simple system > needs which is simply a makefile or script that grabs each consolidation, > sets up the environment, builds and moves onto the next (with appropriate > error handling etc.) > > Aszezo has some idea and how to get this party started I believe, so > probably best for him to make more concrete suggestions? > > If we could get a empty framework in place and running, we could then pop in > each consolidation in turn when ready, and therefore gradually get our > continuous build system working, the particularly troublesome ones don't > have to stop the easier ones. > > A hack-a-thon might make an ideal time for people to get together and push > this all together? > > Thoughts please, > Deano > > > > ___ > oi-dev mailing list > oi-dev@openindiana.org > http://openindiana.org/mailman/listinfo/oi-dev > -- T.J. Yang ___ oi-dev mailing list oi-dev@openindiana.org http://openindiana.org/mailman/listinfo/oi-dev
Re: [oi-dev] Distribution build system
On Wed, Apr 6, 2011 at 10:43 AM, TJ Yang wrote: > Hi, Deano > > I am converting all the consolidations building instruction into xml files. > and this xml file can be playbacked but /opt/TWWfsw/bin/sb > (sb=software build) tool. s/but/by/ > > Very similar following command to create O.I. CD/USB image. except > the abstraction is one level higher. > > pfexec distro_const build ./new_slim_cd_x86.xml > > So for this digitization effort is to generate O.I. sparc image to > provide alternative OS that our Sparc hardware can run freely. > > I don't expect it got adopted by this group since TWW Inc.'s GNU > tool-sets is not well-known. But I am documenting this approach at > R1. > > tj > > R1: http://wiki.openindiana.org/oi/CPAMTWW > > On Wed, Apr 6, 2011 at 7:13 AM, Deano wrote: >> Hello, >> >> The conversation on the Distributed build system stalled on finer points but >> the large idea seemed to have everyone support. A continuous build system, >> that automatically went through all consolidations would be a major step >> forward for the OI development process. >> So lets get things moved on a least for the most basic system we all agree >> would be useful. I looked into some more advanced build systems (Scons, WAF, >> etc.) but tbh all were slightly overkill for what the basic simple system >> needs which is simply a makefile or script that grabs each consolidation, >> sets up the environment, builds and moves onto the next (with appropriate >> error handling etc.) >> >> Aszezo has some idea and how to get this party started I believe, so >> probably best for him to make more concrete suggestions? >> >> If we could get a empty framework in place and running, we could then pop in >> each consolidation in turn when ready, and therefore gradually get our >> continuous build system working, the particularly troublesome ones don't >> have to stop the easier ones. >> >> A hack-a-thon might make an ideal time for people to get together and push >> this all together? >> >> Thoughts please, >> Deano >> >> >> >> ___ >> oi-dev mailing list >> oi-dev@openindiana.org >> http://openindiana.org/mailman/listinfo/oi-dev >> > > > > -- > T.J. Yang > -- T.J. Yang ___ oi-dev mailing list oi-dev@openindiana.org http://openindiana.org/mailman/listinfo/oi-dev
Re: [oi-dev] Distribution build system
Hi TJ, This is great that you are trying TWW tools. I was actually thinking about using a GNU make based system, possibly with few helper scripts. The advantage of using Makefiles is that everyone is familiar with them, incremental builds are possible and also parallel builds are possible. Can you point me to some docs about TWW tools describing what they can do? I have never heard about them :) Thanks, Andrzej On 04/06/11 16:44, TJ Yang wrote: On Wed, Apr 6, 2011 at 10:43 AM, TJ Yang wrote: Hi, Deano I am converting all the consolidations building instruction into xml files. and this xml file can be playbacked but /opt/TWWfsw/bin/sb (sb=software build) tool. s/but/by/ Very similar following command to create O.I. CD/USB image. except the abstraction is one level higher. pfexec distro_const build ./new_slim_cd_x86.xml So for this digitization effort is to generate O.I. sparc image to provide alternative OS that our Sparc hardware can run freely. I don't expect it got adopted by this group since TWW Inc.'s GNU tool-sets is not well-known. But I am documenting this approach at R1. tj R1: http://wiki.openindiana.org/oi/CPAMTWW On Wed, Apr 6, 2011 at 7:13 AM, Deano wrote: Hello, The conversation on the Distributed build system stalled on finer points but the large idea seemed to have everyone support. A continuous build system, that automatically went through all consolidations would be a major step forward for the OI development process. So lets get things moved on a least for the most basic system we all agree would be useful. I looked into some more advanced build systems (Scons, WAF, etc.) but tbh all were slightly overkill for what the basic simple system needs which is simply a makefile or script that grabs each consolidation, sets up the environment, builds and moves onto the next (with appropriate error handling etc.) Aszezo has some idea and how to get this party started I believe, so probably best for him to make more concrete suggestions? If we could get a empty framework in place and running, we could then pop in each consolidation in turn when ready, and therefore gradually get our continuous build system working, the particularly troublesome ones don't have to stop the easier ones. A hack-a-thon might make an ideal time for people to get together and push this all together? Thoughts please, Deano ___ oi-dev mailing list oi-dev@openindiana.org http://openindiana.org/mailman/listinfo/oi-dev -- T.J. Yang ___ oi-dev mailing list oi-dev@openindiana.org http://openindiana.org/mailman/listinfo/oi-dev
Re: [oi-dev] Distribution build system
On Fri, Apr 8, 2011 at 7:04 PM, Andrzej Szeszo wrote: > Hi TJ, > > This is great that you are trying TWW tools. > > I was actually thinking about using a GNU make based system, possibly with > few helper scripts. FREEBSD:make buildworld How about OpenIndiana: make buildworld too ;) > > The advantage of using Makefiles is that everyone is familiar with them, > incremental builds are possible and also parallel builds are possible. > Agree about using make. At work, I do use make on top of TWW sb/pb tools to achieve incremental builds and parallel builds(not actually doing this right now). for parallel/distributed builds, I played with distcc before but end up using ElectricCloud(different name now)[3]. I am hoping OI project can implement C.I. tools to distribute the build(once above is ready). > Can you point me to some docs about TWW tools describing what they can do? I > have never heard about them :) See [2] Reference section. the best doc so far is still the RTFM. each tools has detail doc in its manpages. > > Thanks, > > Andrzej > > [1] http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/makeworld.html [2] http://wiki.openindiana.org/oi/Building+O.I.+by+sb [3] http://www.electric-cloud.com/products/electricaccelerator.php tj > On 04/06/11 16:44, TJ Yang wrote: >> >> On Wed, Apr 6, 2011 at 10:43 AM, TJ Yang wrote: >>> >>> Hi, Deano >>> >>> I am converting all the consolidations building instruction into xml >>> files. >>> and this xml file can be playbacked but /opt/TWWfsw/bin/sb >>> (sb=software build) tool. >> >> s/but/by/ >> >>> Very similar following command to create O.I. CD/USB image. except >>> the abstraction is one level higher. >>> >>> pfexec distro_const build ./new_slim_cd_x86.xml >>> >>> So for this digitization effort is to generate O.I. sparc image to >>> provide alternative OS that our Sparc hardware can run freely. >>> >>> I don't expect it got adopted by this group since TWW Inc.'s GNU >>> tool-sets is not well-known. But I am documenting this approach at >>> R1. >>> >>> tj >>> >>> R1: http://wiki.openindiana.org/oi/CPAMTWW >>> >>> On Wed, Apr 6, 2011 at 7:13 AM, Deano wrote: Hello, The conversation on the Distributed build system stalled on finer points but the large idea seemed to have everyone support. A continuous build system, that automatically went through all consolidations would be a major step forward for the OI development process. So lets get things moved on a least for the most basic system we all agree would be useful. I looked into some more advanced build systems (Scons, WAF, etc.) but tbh all were slightly overkill for what the basic simple system needs which is simply a makefile or script that grabs each consolidation, sets up the environment, builds and moves onto the next (with appropriate error handling etc.) Aszezo has some idea and how to get this party started I believe, so probably best for him to make more concrete suggestions? If we could get a empty framework in place and running, we could then pop in each consolidation in turn when ready, and therefore gradually get our continuous build system working, the particularly troublesome ones don't have to stop the easier ones. A hack-a-thon might make an ideal time for people to get together and push this all together? Thoughts please, Deano ___ oi-dev mailing list oi-dev@openindiana.org http://openindiana.org/mailman/listinfo/oi-dev >>> >>> >>> -- >>> T.J. Yang >>> >> >> > -- T.J. Yang ___ oi-dev mailing list oi-dev@openindiana.org http://openindiana.org/mailman/listinfo/oi-dev
[oi-dev] Distribution build system requirements?
What computer resources are needed to build all the distribution images overnight, i.e. 8 hours, from a local repository? I'm on a 3 Mb/s downlink, so as a practical matter I'm better off building and testing release candidates locally. I'd like to take on the work. It's a few weeks twice a year which I can comfortably manage. I have plenty of professional experience producing internal software releases in major oil companies. I've also done FreeBSD, Debian, Solaris 11 and Solaris 10 installs in the last week. For all the flaws I encountered in the Hipster 2020.10 images, it's still better than the others in many ways. I'm proposing to work on the software to build the installation media with a strong focus on intensive testing of the install process and making it more flexible. The installation process is the new user's first impression. Gparted not working doesn't give a good impression. Nor does, "you can only use 2 TB of your new 5 TB disk". I'd like to fix that. Have Fun! Reg ___ oi-dev mailing list oi-dev@openindiana.org https://openindiana.org/mailman/listinfo/oi-dev
Re: [oi-dev] Distribution build system requirements?
On Tue, Apr 6, 2021 at 12:45 AM Reginald Beardsley via oi-dev < oi-dev@openindiana.org> wrote: > What computer resources are needed to build all the distribution images > overnight, i.e. 8 hours, from a local repository? > On my workstation (4 year old 10 core machine with 48GB RAM) it takes a ~2h if I remember correctly. I do not think it is CPU bound as the work is mainly serialized. > I'm on a 3 Mb/s downlink, so as a practical matter I'm better off building > and testing release candidates locally. > > I'd like to take on the work. It's a few weeks twice a year which I can > comfortably manage. I have plenty of professional experience producing > internal software releases in major oil companies. > > I've also done FreeBSD, Debian, Solaris 11 and Solaris 10 installs in the > last week. For all the flaws I encountered in the Hipster 2020.10 images, > it's still better than the others in many ways. I'm proposing to work on > the software to build the installation media with a strong focus on > intensive testing of the install process and making it more flexible. > Once you have the distro constructor set up on the machine it is fairly easy. I expected the initial setup would take ~30min and some trial and error to modify the vanilla manifests. e.g. to use your own local repository to spin custom images. > > The installation process is the new user's first impression. Gparted not > working doesn't give a good impression. Nor does, "you can only use 2 TB > of your new 5 TB disk". I'd like to fix that. > I know gparted is not great... I attempted to port our patchset to newer versions a long time ago but it was too much work... > > Have Fun! > Reg > > ___ > oi-dev mailing list > oi-dev@openindiana.org > https://openindiana.org/mailman/listinfo/oi-dev > -- --- Praise the Caffeine embeddings ___ oi-dev mailing list oi-dev@openindiana.org https://openindiana.org/mailman/listinfo/oi-dev
Re: [oi-dev] Distribution build system requirements?
On Tuesday, April 13, 2021, 05:19:05 AM CDT, Aurélien Larcher wrote: [snip] I know gparted is not great... I attempted to port our patchset to newer versions a long time ago but it was too much work... The issue is gparted dumps core on the 2020.10 Live Image. It works on the 2017.10 Live Image and the 2020.10 installed system. I'm going to look very closely at format as I think it might serve for EFI/GPT labels more sensibly. I can see no reason for having 128 partitions. And I did manage to get a traditional Sun partition table with 9 slots and an EFI label on a 5 TB disk. IIRC I had to format it in Win 7 and then relabel it with format(1m). I just bought an HP Z840 with 1x 14 core E5-2690 V4 on ebay. I need to add 64 GB of DRAM which may be a bit of a hassle because of the global chip shortage. I'm planning on 4x 4 TB disks in RAIDZ2 configuration. A single slice if booting a RAIDZ2 pool with a disk missing is reliable. Have Fun! Reg ___ oi-dev mailing list oi-dev@openindiana.org https://openindiana.org/mailman/listinfo/oi-dev
Re: [oi-dev] Distribution build system requirements?
If anyone gets around to working on gparted (or an alternative) please consider making it use "megabyte aligned partitions" and ignore so-called "geometry" (like partition editors do on pretty much every other OS does now). On Tue, Apr 13, 2021 at 9:02 AM Reginald Beardsley via oi-dev wrote: > > On Tuesday, April 13, 2021, 05:19:05 AM CDT, Aurélien Larcher > wrote: > > [snip] > > I know gparted is not great... I attempted to port our patchset to newer > versions a long time ago but it was too much work... > > > > The issue is gparted dumps core on the 2020.10 Live Image. It works on the > 2017.10 Live Image and the 2020.10 installed system. I'm going to look very > closely at format as I think it might serve for EFI/GPT labels more sensibly. > I can see no reason for having 128 partitions. And I did manage to get a > traditional Sun partition table with 9 slots and an EFI label on a 5 TB disk. > IIRC I had to format it in Win 7 and then relabel it with format(1m). > > I just bought an HP Z840 with 1x 14 core E5-2690 V4 on ebay. I need to add 64 > GB of DRAM which may be a bit of a hassle because of the global chip > shortage. I'm planning on 4x 4 TB disks in RAIDZ2 configuration. A single > slice if booting a RAIDZ2 pool with a disk missing is reliable. > > Have Fun! > Reg > ___ > oi-dev mailing list > oi-dev@openindiana.org > https://openindiana.org/mailman/listinfo/oi-dev ___ oi-dev mailing list oi-dev@openindiana.org https://openindiana.org/mailman/listinfo/oi-dev
Re: [oi-dev] Distribution build system requirements?
I won't get to it until summer as at the moment outdoor work here is tolerable, but I plan to go over the entire issue of disk labeling and partitioning. There are a huge number of what are retrospectively poor decisions and ancient cruft that needs to be addressed in format(1m). On consideration, I think gparted should go away and the functionality moved to format(1m). Of course, that presumes that the OI/Illumos community are still comfortable with command line tools and do not require a GUI. I just bought an HP Z840 and will be spending the next month or so configuring it to build Illumos and OI in preparation for doing a modest amount of serious work on things that annoy me. I'm not so old as to have dealt with SMD drives, but I had lots of experience with the disk geometry and remember format(1m) responding to a 5400 rpm speed with a "preposterous" error message. I need to do a *lot* of reading about how modern drives behave with respect to alignment. At present it appears to me that we have lie stacked upon lie stacked upon lie. It's time for a bit of truth. Have Fun! Reg On Wednesday, April 14, 2021, 06:36:20 PM CDT, Gordon Ross wrote: If anyone gets around to working on gparted (or an alternative) please consider making it use "megabyte aligned partitions" and ignore so-called "geometry" (like partition editors do on pretty much every other OS does now). On Tue, Apr 13, 2021 at 9:02 AM Reginald Beardsley via oi-dev wrote: > > On Tuesday, April 13, 2021, 05:19:05 AM CDT, Aurélien Larcher > wrote: > > [snip] > > I know gparted is not great... I attempted to port our patchset to newer > versions a long time ago but it was too much work... > > > > The issue is gparted dumps core on the 2020.10 Live Image. It works on the > 2017.10 Live Image and the 2020.10 installed system. I'm going to look very > closely at format as I think it might serve for EFI/GPT labels more sensibly. > I can see no reason for having 128 partitions. And I did manage to get a > traditional Sun partition table with 9 slots and an EFI label on a 5 TB disk. > IIRC I had to format it in Win 7 and then relabel it with format(1m). > > I just bought an HP Z840 with 1x 14 core E5-2690 V4 on ebay. I need to add 64 > GB of DRAM which may be a bit of a hassle because of the global chip > shortage. I'm planning on 4x 4 TB disks in RAIDZ2 configuration. A single > slice if booting a RAIDZ2 pool with a disk missing is reliable. > > Have Fun! > Reg > ___ > oi-dev mailing list > oi-dev@openindiana.org > https://openindiana.org/mailman/listinfo/oi-dev ___ oi-dev mailing list oi-dev@openindiana.org https://openindiana.org/mailman/listinfo/oi-dev
