Re: Taskotron and Cloud Image tests
On Thu, 17 Apr 2014 17:35:13 +0200 Vitaly Kuznetsov wrote: > > Another possible snag is that I want to start locking down network > > access on most if not all of the test clients so that it's less > > possible for user-submitted tasks to go awry and do things they > > shouldn't. This hasn't been done yet, though and it's something > > that we can discuss going forward. > > For valid we'll require two things: > 1) Access to Cloud's (AWS, Openstack, ...) endpoint > 2) SSH to running VM Interfacing with EC2 wasn't a use-case that I was thinking of for network isolation of the taskotron clients, so those plans may change somewhat. The clients aren't isolated yet, so this won't be a problem immediately. > Can we have special dedicated test client for valid? That would make > sense from securitty pov as we need to store cloud access credentials > there. I suppose that we could but I'd really prefer to avoid that if at all possible. Having one "special" client isn't an issue but it does open the door to other task authors asking for the same thing and that will get unmanageable pretty quick. That being said, I'm not sure how to go about managing credentials like that in a secure fashion. This'll require some more thought but suggestions are certainly welcome :) Tim signature.asc Description: PGP signature ___ qa-devel mailing list qa-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/qa-devel
Re: Taskotron and Cloud Image tests
Tim Flink writes: > On Thu, 17 Apr 2014 14:25:03 +0200 > Vitaly Kuznetsov wrote: > >> Tim Flink writes: >> >> >> For RHEL cloud image checks we have special tool called 'valid': >> >> https://github.com/RedHatQE/valid , it can be reused for Fedora as >> >> well. The questions are: how do you see the integration and how can >> >> someone (me?) work on this (non-existent atm) part of Taskotron. >> > >> > For now, I think that the best place to start would be to figure out >> > what's missing from taskotron in order for the integration to >> > happen. >> > - would the cloud image startup be done by valid or in taskotron? >> > - where would the cloud images be run? >> > - what other image work would be done in the task? >> >> Atm 'valid' gets EC2 AMI id as an input and produces test result (yaml >> file with all tests, their steps and results + logs). I think we can >> easily kick off such testing tasks from taskotron. The only missing >> piece in the picture is an 'image uploader': after image was built in >> koji someone needs to upload it as an AMI to EC2 (afaict Dennis >> does that manually now). > > The concern I have with this is around EC2 credentials and accounts. If > I'm understanding you correctly, we would need to have those > credentials stored on the clients and we'd be using EC2 time for > testing Fedora. > > Does someone have budget or account time that could be used for this > in an account that we could use? I wonder if trying to use the fedora > openstack setup be a better route (assuming that valid can work with > openstack). > It can't right now but switching from boto to something like libcloud is already in my backlog. In case we do have such requirement (and Matt's letter says we do) I can increase the priority for the task. > Another possible snag is that I want to start locking down network > access on most if not all of the test clients so that it's less > possible for user-submitted tasks to go awry and do things they > shouldn't. This hasn't been done yet, though and it's something that we > can discuss going forward. For valid we'll require two things: 1) Access to Cloud's (AWS, Openstack, ...) endpoint 2) SSH to running VM Can we have special dedicated test client for valid? That would make sense from securitty pov as we need to store cloud access credentials there. >> 2mattdm: Matt, I remember you telling me that someone is working on >> such uploader. Can you please shed some light on what's the status >> now and what can be done in before f21? >> >> >> I can see two possible approaches: >> >> 1) 'Easy'. We create special type of task (sorry if I'm not that >> >> precise with terminology) which is being executed on static slave >> >> node which runs valid (in black-box mode) on newly uploaded image, >> >> grabs the result and returns it back. >> > >> > I suspect that this will be the best way to get started. The >> > ephemeral client work you mention below is going to be a lot of work >> > and there are a bunch of unknowns (use openstack or not, how >> > isolated should the clients be etc.). If we want to have a shot at >> > having some of the automated cloud tests done for f21, I think that >> > starting with the easiest route would likely be best >> >> Ok, I think that having something ready in f21 timeframe is really >> doable in case releng can automate AMIs uploading process. I'll take a >> look at the runner in libtaskotron and existing tasks examples. > > Cool, let us know if you have any questions or if anything isn't clear. > > Tim -- Vitaly Kuznetsov, Cloud QE ___ qa-devel mailing list qa-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/qa-devel
Re: Taskotron and Cloud Image tests
On Thu, Apr 17, 2014 at 09:11:10AM -0600, Tim Flink wrote: > Does someone have budget or account time that could be used for this > in an account that we could use? I wonder if trying to use the fedora > openstack setup be a better route (assuming that valid can work with > openstack). I think we can use the Fedora community cloud EC2 account for this, but I will double check. If that's not possible, I'll beg for the required budget. We _should_ test on both OpenStack and EC2. -- Matthew Miller-- Fedora Project-- "Tepid change for the somewhat better!" ___ qa-devel mailing list qa-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/qa-devel
Re: Taskotron and Cloud Image tests
On Thu, 17 Apr 2014 14:25:03 +0200 Vitaly Kuznetsov wrote: > Tim Flink writes: > > >> For RHEL cloud image checks we have special tool called 'valid': > >> https://github.com/RedHatQE/valid , it can be reused for Fedora as > >> well. The questions are: how do you see the integration and how can > >> someone (me?) work on this (non-existent atm) part of Taskotron. > > > > For now, I think that the best place to start would be to figure out > > what's missing from taskotron in order for the integration to > > happen. > > - would the cloud image startup be done by valid or in taskotron? > > - where would the cloud images be run? > > - what other image work would be done in the task? > > Atm 'valid' gets EC2 AMI id as an input and produces test result (yaml > file with all tests, their steps and results + logs). I think we can > easily kick off such testing tasks from taskotron. The only missing > piece in the picture is an 'image uploader': after image was built in > koji someone needs to upload it as an AMI to EC2 (afaict Dennis > does that manually now). The concern I have with this is around EC2 credentials and accounts. If I'm understanding you correctly, we would need to have those credentials stored on the clients and we'd be using EC2 time for testing Fedora. Does someone have budget or account time that could be used for this in an account that we could use? I wonder if trying to use the fedora openstack setup be a better route (assuming that valid can work with openstack). Another possible snag is that I want to start locking down network access on most if not all of the test clients so that it's less possible for user-submitted tasks to go awry and do things they shouldn't. This hasn't been done yet, though and it's something that we can discuss going forward. > 2mattdm: Matt, I remember you telling me that someone is working on > such uploader. Can you please shed some light on what's the status > now and what can be done in before f21? > > >> I can see two possible approaches: > >> 1) 'Easy'. We create special type of task (sorry if I'm not that > >> precise with terminology) which is being executed on static slave > >> node which runs valid (in black-box mode) on newly uploaded image, > >> grabs the result and returns it back. > > > > I suspect that this will be the best way to get started. The > > ephemeral client work you mention below is going to be a lot of work > > and there are a bunch of unknowns (use openstack or not, how > > isolated should the clients be etc.). If we want to have a shot at > > having some of the automated cloud tests done for f21, I think that > > starting with the easiest route would likely be best > > Ok, I think that having something ready in f21 timeframe is really > doable in case releng can automate AMIs uploading process. I'll take a > look at the runner in libtaskotron and existing tasks examples. Cool, let us know if you have any questions or if anything isn't clear. Tim signature.asc Description: PGP signature ___ qa-devel mailing list qa-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/qa-devel
Re: Taskotron and Cloud Image tests
Tim Flink writes: >> For RHEL cloud image checks we have special tool called 'valid': >> https://github.com/RedHatQE/valid , it can be reused for Fedora as >> well. The questions are: how do you see the integration and how can >> someone (me?) work on this (non-existent atm) part of Taskotron. > > For now, I think that the best place to start would be to figure out > what's missing from taskotron in order for the integration to happen. > - would the cloud image startup be done by valid or in taskotron? > - where would the cloud images be run? > - what other image work would be done in the task? Atm 'valid' gets EC2 AMI id as an input and produces test result (yaml file with all tests, their steps and results + logs). I think we can easily kick off such testing tasks from taskotron. The only missing piece in the picture is an 'image uploader': after image was built in koji someone needs to upload it as an AMI to EC2 (afaict Dennis does that manually now). 2mattdm: Matt, I remember you telling me that someone is working on such uploader. Can you please shed some light on what's the status now and what can be done in before f21? >> I can see two possible approaches: >> 1) 'Easy'. We create special type of task (sorry if I'm not that >> precise with terminology) which is being executed on static slave >> node which runs valid (in black-box mode) on newly uploaded image, >> grabs the result and returns it back. > > I suspect that this will be the best way to get started. The > ephemeral client work you mention below is going to be a lot of work > and there are a bunch of unknowns (use openstack or not, how isolated > should the clients be etc.). If we want to have a shot at having some > of the automated cloud tests done for f21, I think that starting with > the easiest route would likely be best Ok, I think that having something ready in f21 timeframe is really doable in case releng can automate AMIs uploading process. I'll take a look at the runner in libtaskotron and existing tasks examples. > > Apologies for the delayed response. Looking forward to working with you > on cloud image testing. > No prob) -- Vitaly Kuznetsov ___ qa-devel mailing list qa-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/qa-devel
Re: Taskotron and Cloud Image tests
Vitaly Kuznetsov writes: > Hi Tim & all, > > in Cloud WG we have 'Automatic Smoketests on Image Build' task: > https://fedorahosted.org/cloud/ticket/38 > AFAIK you have somehting similar among future Taskotron goals. It would > be nice if we can collaborate here. By writing this message I just want > to start the conversation :-) > > For RHEL cloud image checks we have special tool called 'valid': > https://github.com/RedHatQE/valid , it can be reused for Fedora as > well. The questions are: how do you see the integration and how can > someone (me?) work on this (non-existent atm) part of Taskotron. > > I can see two possible approaches: > 1) 'Easy'. We create special type of task (sorry if I'm not that precise > with terminology) which is being executed on static slave node which > runs valid (in black-box mode) on newly uploaded image, grabs the result > and returns it back. > > 2) 'Hard'. We work on 'ephemeral task clients' (term from > http://tirfa.com/an-initial-idea-for-taskbot.html). The benefit here is > that in that case we can run any test on a cloud VM. This approach will > require some optimization from the very beginning, at least: > 1. One VM should be user for multiple tests (so several dozens of image > tests won't require several dozens of VMs created) > 2. Tests should be able to 'control' VM (e.g. use cloud-specific > features: attach device, reboot VM,...) - that can be workarounded by > providing cloud credential inside the VM itself but that doesn't sound > that attractive. > > Yesterday I've tried setting up Taskotron (according to > http://roshi.fedorapeople.org/dexy-themed/) on 3 F20 VMs in EC2 and > actually I've failed (some ansible playbooks are out-of-sync with some > git repos, e.g. 'master playbook' is failing with 'stderr: cp: cannot > stat '/home/qaadmin/trigger/jobtrigger.py': No such file or directory' > in 'TASK: [copy fedmsg config]'. I totally understand these instructions > and repos are rather proof-of-concept and can be slightly outdated but > I want to ask about 'minimalistic one-host dev environment' setup > anyway. Can 'Host' be eliminated from the setup completely and can we > merge Master and Slave (in case we'll need it for 'cloud') on one host? > That would help a lot. > > I'm looking forward to hearing from you. Sorry to bug you again about this, am I completely out of sync with whatever is going on? I would really appreciate your comments.. -- Vitaly Kuznetsov ___ qa-devel mailing list qa-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/qa-devel
Taskotron and Cloud Image tests
Hi Tim & all, in Cloud WG we have 'Automatic Smoketests on Image Build' task: https://fedorahosted.org/cloud/ticket/38 AFAIK you have somehting similar among future Taskotron goals. It would be nice if we can collaborate here. By writing this message I just want to start the conversation :-) For RHEL cloud image checks we have special tool called 'valid': https://github.com/RedHatQE/valid , it can be reused for Fedora as well. The questions are: how do you see the integration and how can someone (me?) work on this (non-existent atm) part of Taskotron. I can see two possible approaches: 1) 'Easy'. We create special type of task (sorry if I'm not that precise with terminology) which is being executed on static slave node which runs valid (in black-box mode) on newly uploaded image, grabs the result and returns it back. 2) 'Hard'. We work on 'ephemeral task clients' (term from http://tirfa.com/an-initial-idea-for-taskbot.html). The benefit here is that in that case we can run any test on a cloud VM. This approach will require some optimization from the very beginning, at least: 1. One VM should be user for multiple tests (so several dozens of image tests won't require several dozens of VMs created) 2. Tests should be able to 'control' VM (e.g. use cloud-specific features: attach device, reboot VM,...) - that can be workarounded by providing cloud credential inside the VM itself but that doesn't sound that attractive. Yesterday I've tried setting up Taskotron (according to http://roshi.fedorapeople.org/dexy-themed/) on 3 F20 VMs in EC2 and actually I've failed (some ansible playbooks are out-of-sync with some git repos, e.g. 'master playbook' is failing with 'stderr: cp: cannot stat '/home/qaadmin/trigger/jobtrigger.py': No such file or directory' in 'TASK: [copy fedmsg config]'. I totally understand these instructions and repos are rather proof-of-concept and can be slightly outdated but I want to ask about 'minimalistic one-host dev environment' setup anyway. Can 'Host' be eliminated from the setup completely and can we merge Master and Slave (in case we'll need it for 'cloud') on one host? That would help a lot. I'm looking forward to hearing from you. -- Vitaly Kuznetsov ___ qa-devel mailing list qa-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/qa-devel
