[ https://issues.apache.org/jira/browse/CLOUDSTACK-7143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14121343#comment-14121343 ]
ASF GitHub Bot commented on CLOUDSTACK-7143: -------------------------------------------- GitHub user lsimons opened a pull request: https://github.com/apache/cloudstack/pull/16 CLOUDSTACK-7143: Refactoring of the systemvm build process E-mail thread: http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201407.mbox/%3C7A6CF878-7A28-4D4A-BCD2-0C264F8C90B7%40schubergphilis.com%3E This started out as wanting the systemvm build to take systemvm/patches/debian/{debian,vpn} from the local machine/branch, rather than downloading from the apache git master [1]. In working out how on earth to get veewee to do that cleanly (hint: you can’t, hence resorting to shar usage) I got quite frustrated with the image rebuild times. It so happens that veewee has a --skip-to-postinstall instruction which is _quite_ useful while debugging these scripts. To get that working requires the post install steps to be retryable/convergent. Of course, our existing scripts weren’t set up for that. So I had to add a bunch of tests whether changes had applied already. Which implied a pretty significant refactor. Summarizing this kind of thing is always hard...it’s many little things...the interesting stuff is at the end/bottom, in particular the two main improvements https://github.com/schubergphilis/cloudstack/commit/142d087f6a97f6ac70a858a35d2fe8b638c58cbb When working on the systemvm in isolation, or using vagrant or similar tools, it can be useful to inject a custom SSH key before merging a management server systemvm.iso into it. This option allows that. It should _not_ have effect on management-server- managed vms which always get their SSH keys injected. https://github.com/schubergphilis/cloudstack/commit/e2240eaed18000d4d94dbf6a6e40612db1aeda34 The current build downloads its script from master by fetching a cloudstack tarball. Besides being an unneeded load on the apache git server, this is a problem when working on a branch and wanting to inject a different set of scripts. It also makes it pretty likely that the injected copy of the script will not match what a production release wants, so there is very little chance of not needing to overwrite the scripts. Ideally we would just rsync over some files. However, veewee does not provide an option to do that. In order to keep a 'cleanly veewee-only' build possible, and work with any recent veewee version, in this change we restor to using shar (http://en.wikipedia.org/wiki/Shar) to produce an archive which can execute as a script, which we feed to veewee to execute. In order to avoid having to re-do this cleanup twice, I also ended up merging the systemvm and systemvm64 template definitions, factoring out their small differences by inspecting the os architecture. https://github.com/schubergphilis/cloudstack/commit/f570b3921cd52672f841fc5f99cdd96f9737d629 https://github.com/schubergphilis/cloudstack/commit/50e91217f90fc952182dccac02a5af06ac33fb45 Everything else…well it pretty much falls into two categories: * general code cleanup without functional changes * general code defensiveness to survive various jenkins build scenarios All in all it should help with ongoing maintenance, I think. Most of these commits are now a while old but I wanted to wait with sending this upstream until we had sufficiently tested the systemvms built with this changed approach locally. You can merge this pull request into a Git repository by running: $ git pull https://github.com/schubergphilis/cloudstack feature/systemvm-refactor-for-upstream Alternatively you can review and apply these changes as the patch at: https://github.com/apache/cloudstack/pull/16.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16 ---- commit 49cbb4e20b70a19286c626ac8d9111be8752bc3f Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T07:54:13Z CLOUDSTACK-7143: upgrade systemvm to latest debian stable, 7.6.0. commit 35b1875226201d5923dea57db64bbd789d9ad908 Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T07:55:37Z CLOUDSTACK-7143: split base.sh into its two functions. commit fb258b506964ca339ef85aa92a6217dc12259811 Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T07:57:49Z CLOUDSTACK-7143: move network tuning from cleanup.sh to its own script. commit 50e2c0177b0df04aa80d19264d52c8d55dc02eb3 Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T08:36:11Z CLOUDSTACK-7143: merge systemvm templates, step 1: remove differences commit 03dba3d901997619d9095e33afe4b39a423ac3f6 Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T08:44:38Z CLOUDSTACK-7143: merge systemvm templates, step 2: only one template remains commit afe696edc32dfcaf7eadcb8272f5e1fe31dc32a8 Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T08:52:01Z CLOUDSTACK-7143: remove 'manual' from preseed config. No other changes. commit d7aca05d1a728ffdcecf635cf5e339038ec964e9 Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T08:54:57Z CLOUDSTACK-7143: reformat definition.rb code. No other changes. commit 6b3657744ee14118bd8c1c72c7154b936de2298d Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T08:57:32Z CLOUDSTACK-7143: move add_backports to apt_upgrade.sh commit 751a4d20ba31386f70987e10059328c718bf1d6e Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T08:58:57Z CLOUDSTACK-7143: cleanup apt_upgrade.sh code commit 6129113983a7f5c8c91e98f63d567469d1fcdd22 Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T09:01:15Z CLOUDSTACK-7143: add vagrant_box_build_time file that's standard with current veewee templates commit 58a11d37f6441807a9cf65b7440910ec780fc5ee Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T09:01:56Z CLOUDSTACK-7143: cleanup configure_gruh.sh code Adding a check whether GRUB_TIMEOUT has already been set, and if it has, skip re-running update-grub. commit 82dedcc0f6fc085961409f55f23cfb568f06558a Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T09:09:41Z CLOUDSTACK-7143: split login config out from postinstall.sh commit 3bf9ddf208593a95843ebb728b735bdb32f91ef0 Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T09:10:59Z CLOUDSTACK-7143: cleanup configure_login.sh code Had to change various things to make this code re-entrant. In particular, the sed-based manipulation of /etc/sudoers is gone and replaced with a simpler, minimal (but compatible) sudoers file. Remove the sshd_config tuning since sshd_config is overwritten when we apply the cloud_scripts overlay (from build.sh). commit ae65bf132457313b6ec826bbfde78f939337ab88 Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T09:15:23Z CLOUDSTACK-7143: split locale config into its own file commit d87d60a1a8dfd887d4b37f23e89cee8354538d0a Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T09:16:48Z CLOUDSTACK-7143: cleanup configure_locale.sh code commit 77677a3318f14fb51cdf4574717269adeaef906e Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T09:19:03Z CLOUDSTACK-7143: move fix_acpid to its own file commit db8eb125caef9b18e3954d70ba024a426651c085 Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T09:19:48Z CLOUDSTACK-7143: cleanup configure_acpid.sh code commit 5003845b77abb97ba63d3cd596b0e946d4532411 Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T09:21:03Z CLOUDSTACK-7143: cleanup cleanup.sh code commit 6e714dc5cd231c9a96b62e304d292017a33be25c Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T09:27:16Z CLOUDSTACK-7143: split connection tracking config into its own file commit 0addd5a1bf0f6810e901f2b4261c4c5a0ec113e6 Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T09:41:37Z CLOUDSTACK-7143: move networking logic from postinstall.sh to configure_networking.sh commit fc87fc4a9ed0b536789500eba573156cadfc6383 Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T09:42:27Z CLOUDSTACK-7143: cleanup configure_networking.sh code commit 95d77c2a4674b09546c436beeb1fdef2534a1eee Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T09:46:02Z CLOUDSTACK-7143: move install_packages from postinstall.sh to its own file commit cdb30cc59d0bbe90c03d6a1a56c93570f97f2947 Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T10:02:13Z CLOUDSTACK-7143: consider vhdutil a software package commit 0e62aab3507e43eaf7598bf3f41f76cced97524e Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T10:07:53Z CLOUDSTACK-7143: reorg of install_systemvm_packages.sh (no logic changes, yet) commit 3ac947fa36cd20a9a7bcd9f467aff53f9af71ace Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T10:17:06Z CLOUDSTACK-7143: collapse apt-get install onto one line In theory this _could_ have changed behavior (apt coming up with a different solution, or one of the packages configuring a new apt repository), but in my testing, the end result is the same. commit 207c50aa20b92a99783fcedb0ac51fb2ec112de6 Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T10:28:32Z CLOUDSTACK-7143: rename postinstall.sh to represent what remains commit b9f606b4f9ab8943533a7b6d3ca43fecc3cdf5b2 Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T11:03:56Z CLOUDSTACK-7143: cleanup configure_systemvm_services.sh code commit 30fe938af9e958972bceaea60275c6b2dfada848 Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T11:06:34Z CLOUDSTACK-7143: tune timeouts to expect about <=~60 min build commit 15673808c7c9500b44e715a3f28f8b21aba0311a Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T11:08:35Z CLOUDSTACK-7143: cleanup zerodisk.sh code commit 44a888aa1a9d8f5a14699b87091ef2e34520208e Author: Leo Simons <lsim...@schubergphilis.com> Date: 2014-07-21T11:09:14Z CLOUDSTACK-7143: cleanup configure_conntrack.sh code ---- > Refactor systemvm build scripts to be easier to test > ---------------------------------------------------- > > Key: CLOUDSTACK-7143 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7143 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: SystemVM > Reporter: Leo Simons > Fix For: Future > > > The veewee-wrapping build code could do with some love. > E-mail thread: > http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201407.mbox/%3C7A6CF878-7A28-4D4A-BCD2-0C264F8C90B7%40schubergphilis.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)