[jira] [Commented] (CLOUDSTACK-7143) Refactor systemvm build scripts to be easier to test
[ https://issues.apache.org/jira/browse/CLOUDSTACK-7143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14145442#comment-14145442 ] ASF GitHub Bot commented on CLOUDSTACK-7143: Github user lsimons commented on the pull request: https://github.com/apache/cloudstack/pull/16#issuecomment-56589622 Thanks for all the help Rohit! I'm sure to most people some bash script rearrangement looks boring, but nevermind that, to me it just feels good to sumbit more than just simple one-line patches to an apache project after such a long time :-) Refactor systemvm build scripts to be easier to test Key: CLOUDSTACK-7143 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7143 Project: CloudStack Issue Type: Bug Security Level: Public(Anyone can view this level - this is the default.) Components: SystemVM Reporter: Leo Simons Fix For: Future The veewee-wrapping build code could do with some love. E-mail thread: http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201407.mbox/%3C7A6CF878-7A28-4D4A-BCD2-0C264F8C90B7%40schubergphilis.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7143) Refactor systemvm build scripts to be easier to test
[ https://issues.apache.org/jira/browse/CLOUDSTACK-7143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14143184#comment-14143184 ] ASF GitHub Bot commented on CLOUDSTACK-7143: Github user bhaisaab commented on the pull request: https://github.com/apache/cloudstack/pull/16#issuecomment-56371520 Hi Leo, I spent last few hours in debugging systemvms. I tested the built systemvms from http://jenkins.buildacloud.org/job/systemvm-refactor-CLOUDSTACK-7143/9/ on KVM using both master and 4.4 branch as my systemvms were unable to connect to mgmt server. At first, I was getting KVM clock error exceptions which I solved by disabling it from agent.properties (http://cjkz.wordpress.com/2014/08/08/apache-cloudstack-kvmclock-error/). So after spending last few hours debugging them, I see the following from VNC/KVM:  It looks like the disk is corrupt for some reason, so to test the image I used the new KVM template to deploy a VM on another CloudStack deployment which perfectly worked for me. So, I'm guessing something in the KVM iso/patching process is messing with the systemvm. Now every time CloudStack tried to do HA and re-deploys systemvms I get the above on my test environment. Can you please try this at your end using KVM or Xen and let me know if it works for you? Refactor systemvm build scripts to be easier to test Key: CLOUDSTACK-7143 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7143 Project: CloudStack Issue Type: Bug Security Level: Public(Anyone can view this level - this is the default.) Components: SystemVM Reporter: Leo Simons Fix For: Future The veewee-wrapping build code could do with some love. E-mail thread: http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201407.mbox/%3C7A6CF878-7A28-4D4A-BCD2-0C264F8C90B7%40schubergphilis.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7143) Refactor systemvm build scripts to be easier to test
[ https://issues.apache.org/jira/browse/CLOUDSTACK-7143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14143202#comment-14143202 ] ASF GitHub Bot commented on CLOUDSTACK-7143: Github user bhaisaab commented on the pull request: https://github.com/apache/cloudstack/pull/16#issuecomment-56373578 I manually ran fsck on failing partitions of the systemvm and rebooted, then it worked for me.  @lsimons I think the issue is probably how we're partitioning and if we're cleanly shutting down the VM before exporting it in various formats? I think once you may help fix it, it should be good to merge. Refactor systemvm build scripts to be easier to test Key: CLOUDSTACK-7143 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7143 Project: CloudStack Issue Type: Bug Security Level: Public(Anyone can view this level - this is the default.) Components: SystemVM Reporter: Leo Simons Fix For: Future The veewee-wrapping build code could do with some love. E-mail thread: http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201407.mbox/%3C7A6CF878-7A28-4D4A-BCD2-0C264F8C90B7%40schubergphilis.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7143) Refactor systemvm build scripts to be easier to test
[ https://issues.apache.org/jira/browse/CLOUDSTACK-7143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14143270#comment-14143270 ] ASF GitHub Bot commented on CLOUDSTACK-7143: Github user lsimons commented on the pull request: https://github.com/apache/cloudstack/pull/16#issuecomment-56385199 thanks for testing rohit. We've never seen this (with xen). If you look at the error, ``` /dev/vda5: Superblock last mount time (Fri Sep 19 14:54...) now = Mon Sep 15 20:34... is in the future ``` the fsck is unhappy with the last recorded mount being sep 19 (last friday, which is correct, i.e. it matches with when we ran the build on jenkins) because the date of the system as its booting is in the past, sep 15. So somehow the system clock of the VM is wrong...? Either during the build process the time was moved backward some time after mounting the filesystem, or during the boot process time moved backward, or ... ? I've checked and the date of the build slave where the image was baked is correct and managed with ntp so very likely was always correct...so I really don't see how the build could result in that timestamp existing anywhere. Can you check the date of your KVM host? Or perhaps if you have an NTP server with a wrong date? Refactor systemvm build scripts to be easier to test Key: CLOUDSTACK-7143 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7143 Project: CloudStack Issue Type: Bug Security Level: Public(Anyone can view this level - this is the default.) Components: SystemVM Reporter: Leo Simons Fix For: Future The veewee-wrapping build code could do with some love. E-mail thread: http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201407.mbox/%3C7A6CF878-7A28-4D4A-BCD2-0C264F8C90B7%40schubergphilis.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7143) Refactor systemvm build scripts to be easier to test
[ https://issues.apache.org/jira/browse/CLOUDSTACK-7143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14143285#comment-14143285 ] ASF GitHub Bot commented on CLOUDSTACK-7143: Github user lsimons commented on the pull request: https://github.com/apache/cloudstack/pull/16#issuecomment-56387394 sorry, nevermind the last comment. From the jenkins build log: ``` [2014-09-19 14:57:59] INFO: stoppping all virtualbox vms for jenkins + bundle exec ./vbox_vm_clean.rb VBoxManage controlvm systemvmtemplate poweroff VBoxManage: error: Invalid machine state: PoweredOff (must be Running, Paused or Stuck) VBoxManage: error: Details: code VBOX_E_INVALID_VM_STATE (0x80bb0002), component Console, interface IConsole, callee nsISupports VBoxManage: error: Context: PowerDown(progress.asOutParam()) at line 222 of file VBoxManageControlVM.cpp VBoxManage controlvm systemvm64template poweroff VBoxManage: error: Invalid machine state: PoweredOff (must be Running, Paused or Stuck) VBoxManage: error: Details: code VBOX_E_INVALID_VM_STATE (0x80bb0002), component Console, interface IConsole, callee nsISupports VBoxManage: error: Context: PowerDown(progress.asOutParam()) at line 222 of file VBoxManageControlVM.cpp kill -SIGKILL 19324 kill -SIGTERM 19324 ./vbox_vm_clean.rb:49:in `kill' ./vbox_vm_clean.rb:49:in `block (2 levels) in main' /usr/local/rvm/gems/ruby-1.9.3-p547/gems/sys-proctable-0.9.4-universal-linux/lib/linux/sys/proctable.rb:237:in `block in ps' /usr/local/rvm/gems/ruby-1.9.3-p547/gems/sys-proctable-0.9.4-universal-linux/lib/linux/sys/proctable.rb:110:in `foreach' /usr/local/rvm/gems/ruby-1.9.3-p547/gems/sys-proctable-0.9.4-universal-linux/lib/linux/sys/proctable.rb:110:in `ps' ./vbox_vm_clean.rb:31:in `block in main' ./vbox_vm_clean.rb:14:in `each' ./vbox_vm_clean.rb:14:in `main' VBoxManage controlvm systemvm64template-systemvm-refactor-for-upstream-9 poweroff VBoxManage: error: Invalid machine state: Aborted (must be Running, Paused or Stuck) VBoxManage: error: Details: code VBOX_E_INVALID_VM_STATE (0x80bb0002), component Console, interface IConsole, callee nsISupports VBoxManage: error: Context: PowerDown(progress.asOutParam()) at line 222 of file VBoxManageControlVM.cpp ``` The lines ```bash kill -SIGKILL 19324 kill -SIGTERM 19324 ``` mean that we forcibly shut down a virtualbox process, probably the one that was running the systemvm that was built. This means that the line further up ```bash bundle exec veewee vbox halt systemvm64template-systemvm-refactor-for-upstream-9 ``` did not succeed in fully shutting down the new machine, and then the next attempt to cleanly power off ```bash VBoxManage controlvm systemvm64template-systemvm-refactor-for-upstream-9 poweroff ``` also failed, and so we don't umount cleanly, and so stuff is broken. FWIW, a successful build has something more like ```bash VBoxManage controlvm systemvmtemplate-systemvm-persistent-config-4.5.0.78 poweroff 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%``` ``` I'll investigate what went wrong. We should probably also just fail the build if the poweroff fails rather than attempting a 'kill'. Refactor systemvm build scripts to be easier to test Key: CLOUDSTACK-7143 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7143 Project: CloudStack Issue Type: Bug Security Level: Public(Anyone can view this level - this is the default.) Components: SystemVM Reporter: Leo Simons Fix For: Future The veewee-wrapping build code could do with some love. E-mail thread: http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201407.mbox/%3C7A6CF878-7A28-4D4A-BCD2-0C264F8C90B7%40schubergphilis.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7143) Refactor systemvm build scripts to be easier to test
[ https://issues.apache.org/jira/browse/CLOUDSTACK-7143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14143293#comment-14143293 ] ASF GitHub Bot commented on CLOUDSTACK-7143: Github user bhaisaab commented on the pull request: https://github.com/apache/cloudstack/pull/16#issuecomment-56388141 Thanks for checking the logs, the time on KVM host was today (22 sep). When I ran fsck it found and fixed couple of corrupt inodes. I think the force shutdown is the only issue we need to fix. After manually fixing the disk volumes I was able to deploy vms. Let me know when you've fixed the vbox shutdown/export issue, I'll retest it and we should be able to merge your PR to master. Cheers. Refactor systemvm build scripts to be easier to test Key: CLOUDSTACK-7143 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7143 Project: CloudStack Issue Type: Bug Security Level: Public(Anyone can view this level - this is the default.) Components: SystemVM Reporter: Leo Simons Fix For: Future The veewee-wrapping build code could do with some love. E-mail thread: http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201407.mbox/%3C7A6CF878-7A28-4D4A-BCD2-0C264F8C90B7%40schubergphilis.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7143) Refactor systemvm build scripts to be easier to test
[ https://issues.apache.org/jira/browse/CLOUDSTACK-7143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14143398#comment-14143398 ] ASF GitHub Bot commented on CLOUDSTACK-7143: Github user lsimons commented on the pull request: https://github.com/apache/cloudstack/pull/16#issuecomment-56402358 Hey Rohit, looks like that did it. Can you try the outputs from http://jenkins.buildacloud.org/job/systemvm-refactor-CLOUDSTACK-7143/10/console when it finishes archiving? Cheers! Refactor systemvm build scripts to be easier to test Key: CLOUDSTACK-7143 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7143 Project: CloudStack Issue Type: Bug Security Level: Public(Anyone can view this level - this is the default.) Components: SystemVM Reporter: Leo Simons Fix For: Future The veewee-wrapping build code could do with some love. E-mail thread: http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201407.mbox/%3C7A6CF878-7A28-4D4A-BCD2-0C264F8C90B7%40schubergphilis.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7143) Refactor systemvm build scripts to be easier to test
[ https://issues.apache.org/jira/browse/CLOUDSTACK-7143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14143659#comment-14143659 ] ASF GitHub Bot commented on CLOUDSTACK-7143: Github user bhaisaab commented on the pull request: https://github.com/apache/cloudstack/pull/16#issuecomment-56425994  Good news! It works for me now. I've tested it against 4.4 branch with KVM basic zone + SG and it LGTM. I'll go ahead with merging it to master now. Thanks for you work Leo! Refactor systemvm build scripts to be easier to test Key: CLOUDSTACK-7143 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7143 Project: CloudStack Issue Type: Bug Security Level: Public(Anyone can view this level - this is the default.) Components: SystemVM Reporter: Leo Simons Fix For: Future The veewee-wrapping build code could do with some love. E-mail thread: http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201407.mbox/%3C7A6CF878-7A28-4D4A-BCD2-0C264F8C90B7%40schubergphilis.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7143) Refactor systemvm build scripts to be easier to test
[ https://issues.apache.org/jira/browse/CLOUDSTACK-7143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14143689#comment-14143689 ] ASF subversion and git services commented on CLOUDSTACK-7143: - Commit ba009ed51a12a8ff3a31b150d488bd632bc10803 in cloudstack's branch refs/heads/master from [~lsimons] [ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=ba009ed ] CLOUDSTACK-7143: attempt to be safer when cleaning up Refactor systemvm build scripts to be easier to test Key: CLOUDSTACK-7143 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7143 Project: CloudStack Issue Type: Bug Security Level: Public(Anyone can view this level - this is the default.) Components: SystemVM Reporter: Leo Simons Fix For: Future The veewee-wrapping build code could do with some love. E-mail thread: http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201407.mbox/%3C7A6CF878-7A28-4D4A-BCD2-0C264F8C90B7%40schubergphilis.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7143) Refactor systemvm build scripts to be easier to test
[ https://issues.apache.org/jira/browse/CLOUDSTACK-7143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14143687#comment-14143687 ] ASF subversion and git services commented on CLOUDSTACK-7143: - Commit 64f4fb1af8efafd8f5548cc19c09d3c8fab3402e in cloudstack's branch refs/heads/master from [~lsimons] [ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=64f4fb1 ] CLOUDSTACK-7143: note shar dependency Refactor systemvm build scripts to be easier to test Key: CLOUDSTACK-7143 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7143 Project: CloudStack Issue Type: Bug Security Level: Public(Anyone can view this level - this is the default.) Components: SystemVM Reporter: Leo Simons Fix For: Future The veewee-wrapping build code could do with some love. E-mail thread: http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201407.mbox/%3C7A6CF878-7A28-4D4A-BCD2-0C264F8C90B7%40schubergphilis.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7143) Refactor systemvm build scripts to be easier to test
[ https://issues.apache.org/jira/browse/CLOUDSTACK-7143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14143686#comment-14143686 ] ASF subversion and git services commented on CLOUDSTACK-7143: - Commit 362946c52e21b70b42b716c8d60431432223f22d in cloudstack's branch refs/heads/master from [~lsimons] [ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=362946c ] CLOUDSTACK-7143: don't call rvm from build It is simpler to expect that rvm setup is done outside of this build. The buildacloud.org jenkins has rvm installed/enabled by default so does not invoke rvm. Refactor systemvm build scripts to be easier to test Key: CLOUDSTACK-7143 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7143 Project: CloudStack Issue Type: Bug Security Level: Public(Anyone can view this level - this is the default.) Components: SystemVM Reporter: Leo Simons Fix For: Future The veewee-wrapping build code could do with some love. E-mail thread: http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201407.mbox/%3C7A6CF878-7A28-4D4A-BCD2-0C264F8C90B7%40schubergphilis.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7143) Refactor systemvm build scripts to be easier to test
[ https://issues.apache.org/jira/browse/CLOUDSTACK-7143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14143688#comment-14143688 ] ASF subversion and git services commented on CLOUDSTACK-7143: - Commit e43e08389328aa45689f184ec34509437c11c3be in cloudstack's branch refs/heads/master from [~lsimons] [ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=e43e083 ] CLOUDSTACK-7143: protect against old ruby Refactor systemvm build scripts to be easier to test Key: CLOUDSTACK-7143 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7143 Project: CloudStack Issue Type: Bug Security Level: Public(Anyone can view this level - this is the default.) Components: SystemVM Reporter: Leo Simons Fix For: Future The veewee-wrapping build code could do with some love. E-mail thread: http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201407.mbox/%3C7A6CF878-7A28-4D4A-BCD2-0C264F8C90B7%40schubergphilis.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7143) Refactor systemvm build scripts to be easier to test
[ https://issues.apache.org/jira/browse/CLOUDSTACK-7143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14143692#comment-14143692 ] ASF GitHub Bot commented on CLOUDSTACK-7143: Github user bhaisaab commented on the pull request: https://github.com/apache/cloudstack/pull/16#issuecomment-56429426 I've rebased your pull request against latest master, fixed some conflicts and merged to latest master. Please check the merge and send any other changes/fixes accordingly. Thanks. Refactor systemvm build scripts to be easier to test Key: CLOUDSTACK-7143 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7143 Project: CloudStack Issue Type: Bug Security Level: Public(Anyone can view this level - this is the default.) Components: SystemVM Reporter: Leo Simons Fix For: Future The veewee-wrapping build code could do with some love. E-mail thread: http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201407.mbox/%3C7A6CF878-7A28-4D4A-BCD2-0C264F8C90B7%40schubergphilis.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7143) Refactor systemvm build scripts to be easier to test
[ https://issues.apache.org/jira/browse/CLOUDSTACK-7143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14143695#comment-14143695 ] ASF GitHub Bot commented on CLOUDSTACK-7143: Github user asfgit closed the pull request at: https://github.com/apache/cloudstack/pull/16 Refactor systemvm build scripts to be easier to test Key: CLOUDSTACK-7143 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7143 Project: CloudStack Issue Type: Bug Security Level: Public(Anyone can view this level - this is the default.) Components: SystemVM Reporter: Leo Simons Fix For: Future The veewee-wrapping build code could do with some love. E-mail thread: http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201407.mbox/%3C7A6CF878-7A28-4D4A-BCD2-0C264F8C90B7%40schubergphilis.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7143) Refactor systemvm build scripts to be easier to test
[ https://issues.apache.org/jira/browse/CLOUDSTACK-7143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14141175#comment-14141175 ] ASF GitHub Bot commented on CLOUDSTACK-7143: Github user bhaisaab commented on the pull request: https://github.com/apache/cloudstack/pull/16#issuecomment-56225895 Looks like the build has passed now: http://jenkins.buildacloud.org/job/systemvm-refactor-CLOUDSTACK-7143/9/ I'll test it against KVM and if it works I'll go ahead and merge it. Thanks. Refactor systemvm build scripts to be easier to test Key: CLOUDSTACK-7143 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7143 Project: CloudStack Issue Type: Bug Security Level: Public(Anyone can view this level - this is the default.) Components: SystemVM Reporter: Leo Simons Fix For: Future The veewee-wrapping build code could do with some love. E-mail thread: http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201407.mbox/%3C7A6CF878-7A28-4D4A-BCD2-0C264F8C90B7%40schubergphilis.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7143) Refactor systemvm build scripts to be easier to test
[ https://issues.apache.org/jira/browse/CLOUDSTACK-7143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14136967#comment-14136967 ] ASF GitHub Bot commented on CLOUDSTACK-7143: Github user lsimons commented on the pull request: https://github.com/apache/cloudstack/pull/16#issuecomment-55866128 Hey Rohit, like I thought, a ruby issue.I'm surprised RVM isn't enabled already since the build was already using it (I think...). The fix should be along the lines of http://rvm.io/integration/jenkins . The other part of the fix is making sure that the inline shell scripts act as login scripts so that rvm is loaded, i.e. they need to start with ``` #!/bin/bash -l ``` Refactor systemvm build scripts to be easier to test Key: CLOUDSTACK-7143 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7143 Project: CloudStack Issue Type: Bug Security Level: Public(Anyone can view this level - this is the default.) Components: SystemVM Reporter: Leo Simons Fix For: Future The veewee-wrapping build code could do with some love. E-mail thread: http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201407.mbox/%3C7A6CF878-7A28-4D4A-BCD2-0C264F8C90B7%40schubergphilis.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7143) Refactor systemvm build scripts to be easier to test
[ https://issues.apache.org/jira/browse/CLOUDSTACK-7143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14136986#comment-14136986 ] ASF GitHub Bot commented on CLOUDSTACK-7143: Github user bhaisaab commented on the pull request: https://github.com/apache/cloudstack/pull/16#issuecomment-55868179 Leo, From the jenkins job log [1] I see that the newly refactored build script is trying to setup ruby [2] and failing because of this. The jenkins job [1] already has rvm setup as I simply cloned it from previous systemvm jobs. Can you look at the job and fix it? If you don't have an account on [1], can may request access to Hugo, Edison, Chip and other PMC members. [1] http://jenkins.buildacloud.org/job/systemvm-refactor-CLOUDSTACK-7143/3/console [2] https://github.com/schubergphilis/cloudstack/blob/feature/systemvm-refactor-for-upstream/tools/appliance/build.sh#L272 Refactor systemvm build scripts to be easier to test Key: CLOUDSTACK-7143 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7143 Project: CloudStack Issue Type: Bug Security Level: Public(Anyone can view this level - this is the default.) Components: SystemVM Reporter: Leo Simons Fix For: Future The veewee-wrapping build code could do with some love. E-mail thread: http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201407.mbox/%3C7A6CF878-7A28-4D4A-BCD2-0C264F8C90B7%40schubergphilis.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7143) Refactor systemvm build scripts to be easier to test
[ https://issues.apache.org/jira/browse/CLOUDSTACK-7143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14133740#comment-14133740 ] ASF GitHub Bot commented on CLOUDSTACK-7143: Github user bhaisaab commented on the pull request: https://github.com/apache/cloudstack/pull/16#issuecomment-55569445 Hi, I'll try to setup a build job for this PR and see if we can merge it on master soon after testing new builds. Refactor systemvm build scripts to be easier to test Key: CLOUDSTACK-7143 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7143 Project: CloudStack Issue Type: Bug Security Level: Public(Anyone can view this level - this is the default.) Components: SystemVM Reporter: Leo Simons Fix For: Future The veewee-wrapping build code could do with some love. E-mail thread: http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201407.mbox/%3C7A6CF878-7A28-4D4A-BCD2-0C264F8C90B7%40schubergphilis.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7143) Refactor systemvm build scripts to be easier to test
[ https://issues.apache.org/jira/browse/CLOUDSTACK-7143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14133865#comment-14133865 ] ASF GitHub Bot commented on CLOUDSTACK-7143: Github user bhaisaab commented on the pull request: https://github.com/apache/cloudstack/pull/16#issuecomment-55586795 I've added a new build job for this PR: http://jenkins.buildacloud.org/job/systemvm-refactor-CLOUDSTACK-7143/ Can you check its build commands, and/or update the README.md file on how to build appliances using build.sh script Refactor systemvm build scripts to be easier to test Key: CLOUDSTACK-7143 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7143 Project: CloudStack Issue Type: Bug Security Level: Public(Anyone can view this level - this is the default.) Components: SystemVM Reporter: Leo Simons Fix For: Future The veewee-wrapping build code could do with some love. E-mail thread: http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201407.mbox/%3C7A6CF878-7A28-4D4A-BCD2-0C264F8C90B7%40schubergphilis.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7143) Refactor systemvm build scripts to be easier to test
[
https://issues.apache.org/jira/browse/CLOUDSTACK-7143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121343#comment-14121343
]
ASF GitHub Bot commented on CLOUDSTACK-7143:
GitHub user lsimons opened a pull request:
https://github.com/apache/cloudstack/pull/16
CLOUDSTACK-7143: Refactoring of the systemvm build process
E-mail thread:
http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201407.mbox/%3C7A6CF878-7A28-4D4A-BCD2-0C264F8C90B7%40schubergphilis.com%3E
This started out as wanting the systemvm build to take
systemvm/patches/debian/{debian,vpn} from the local machine/branch,
rather than downloading from the apache git master [1]. In working out
how on earth to get veewee to do that cleanly (hint: you can’t, hence
resorting to shar usage) I got quite frustrated with the image rebuild
times.
It so happens that veewee has a --skip-to-postinstall instruction which
is _quite_ useful while debugging these scripts. To get that working
requires the post install steps to be retryable/convergent. Of course,
our existing scripts weren’t set up for that. So I had to add a bunch
of tests whether changes had applied already. Which implied a pretty
significant refactor.
Summarizing this kind of thing is always hard...it’s many little
things...the interesting stuff is at the end/bottom, in particular
the two main improvements
https://github.com/schubergphilis/cloudstack/commit/142d087f6a97f6ac70a858a35d2fe8b638c58cbb
When working on the systemvm in isolation, or using vagrant or
similar tools, it can be useful to inject a custom SSH key before
merging a management server systemvm.iso into it. This option
allows that. It should _not_ have effect on management-server-
managed vms which always get their SSH keys injected.
https://github.com/schubergphilis/cloudstack/commit/e2240eaed18000d4d94dbf6a6e40612db1aeda34
The current build downloads its script from master by fetching a
cloudstack tarball. Besides being an unneeded load on the apache
git server, this is a problem when working on a branch and
wanting to inject a different set of scripts. It also makes it
pretty likely that the injected copy of the script will not match
what a production release wants, so there is very little chance of
not needing to overwrite the scripts.
Ideally we would just rsync over some files. However, veewee does
not provide an option to do that. In order to keep a 'cleanly
veewee-only' build possible, and work with any recent veewee
version, in this change we restor to using shar
(http://en.wikipedia.org/wiki/Shar) to produce an archive which
can execute as a script, which we feed to veewee to execute.
In order to avoid having to re-do this cleanup twice, I also ended up
merging the systemvm and systemvm64 template definitions, factoring out
their small differences by inspecting the os architecture.
https://github.com/schubergphilis/cloudstack/commit/f570b3921cd52672f841fc5f99cdd96f9737d629
https://github.com/schubergphilis/cloudstack/commit/50e91217f90fc952182dccac02a5af06ac33fb45
Everything else…well it pretty much falls into two categories:
* general code cleanup without functional changes
* general code defensiveness to survive various jenkins build
scenarios
All in all it should help with ongoing maintenance, I think.
Most of these commits are now a while old but I wanted to wait with
sending this upstream until we had sufficiently tested the systemvms
built with this changed approach locally.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/schubergphilis/cloudstack
feature/systemvm-refactor-for-upstream
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/cloudstack/pull/16.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16
commit 49cbb4e20b70a19286c626ac8d9111be8752bc3f
Author: Leo Simons lsim...@schubergphilis.com
Date: 2014-07-21T07:54:13Z
CLOUDSTACK-7143: upgrade systemvm to latest debian stable, 7.6.0.
commit 35b1875226201d5923dea57db64bbd789d9ad908
Author: Leo Simons lsim...@schubergphilis.com
Date: 2014-07-21T07:55:37Z
CLOUDSTACK-7143: split base.sh into its two functions.
commit fb258b506964ca339ef85aa92a6217dc12259811
Author: Leo Simons lsim...@schubergphilis.com
Date: 2014-07-21T07:57:49Z
CLOUDSTACK-7143: move network tuning from cleanup.sh to its own script.
commit 50e2c0177b0df04aa80d19264d52c8d55dc02eb3
Author: Leo Simons
[jira] [Commented] (CLOUDSTACK-7143) Refactor systemvm build scripts to be easier to test
[ https://issues.apache.org/jira/browse/CLOUDSTACK-7143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121425#comment-14121425 ] ASF GitHub Bot commented on CLOUDSTACK-7143: Github user bhaisaab commented on the pull request: https://github.com/apache/cloudstack/pull/16#issuecomment-54487736 Hi Leo, thanks for the PR. I see you've refactored the 32 and 64 bit building scripts to one which is great! I'll be able to test it next week and help you get it fixed if needed and merged. Meanwhile you may ask Hugo to set a test build job for testing the appliances. And some basic tests using those built systemvm templates both 32 and 64 bits. Refactor systemvm build scripts to be easier to test Key: CLOUDSTACK-7143 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7143 Project: CloudStack Issue Type: Bug Security Level: Public(Anyone can view this level - this is the default.) Components: SystemVM Reporter: Leo Simons Fix For: Future The veewee-wrapping build code could do with some love. E-mail thread: http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201407.mbox/%3C7A6CF878-7A28-4D4A-BCD2-0C264F8C90B7%40schubergphilis.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7143) Refactor systemvm build scripts to be easier to test
[ https://issues.apache.org/jira/browse/CLOUDSTACK-7143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121440#comment-14121440 ] ASF GitHub Bot commented on CLOUDSTACK-7143: Github user lsimons commented on the pull request: https://github.com/apache/cloudstack/pull/16#issuecomment-54490782 Hey Rohit, thanks for reviewing! We actually have a new vagrant-based component test setup for the systemvm to contribute, too; I'm working on extracting that now from the schubergphilis feature/systemvm-persistent-config branch. Our internal jenkins setup is already running those tests, as well as wider-scale marvin-driven integration tests that use these new-style systemvms. That's why I think this is ready for upstream: I'm pretty confident this stuff all still works properly.tough of course that _really_ _really_ needs careful verification outside of our assumptions/environment. I'll try and work with Hugo to port those jenkins builds to buildacloud.org...hope he has time for it :) Refactor systemvm build scripts to be easier to test Key: CLOUDSTACK-7143 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7143 Project: CloudStack Issue Type: Bug Security Level: Public(Anyone can view this level - this is the default.) Components: SystemVM Reporter: Leo Simons Fix For: Future The veewee-wrapping build code could do with some love. E-mail thread: http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201407.mbox/%3C7A6CF878-7A28-4D4A-BCD2-0C264F8C90B7%40schubergphilis.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
