[yocto] [PATCH 12/13] REAMDE-GOOGLE-CLOUD: Update documentation to be more specific.
Signed-off-by: California Sullivan
---
README-GOOGLE-CLOUD | 46 ++
1 file changed, 30 insertions(+), 16 deletions(-)
diff --git a/README-GOOGLE-CLOUD b/README-GOOGLE-CLOUD
index 30bf5b6..34c80a9 100644
--- a/README-GOOGLE-CLOUD
+++ b/README-GOOGLE-CLOUD
@@ -6,7 +6,7 @@ limitations.
First of all, Google's gcloud command line tool is required for
all functions (see https://developers.google.com/cloud/sdk/gcloud/).
-Next, the controller system is required to be authenticated with Google's cloud
+Next, the controller machine is required to be authenticated with Google's
cloud
system (see https://developers.google.com/cloud/sdk/gcloud/reference/auth/).
Third, the workers must be on a network which has the worker port
@@ -21,19 +21,29 @@ gcloud compute firewall-rules autobuilder-allow --network
autobuilder-network \
This sets up a network that is suitable for both controllers and workers, as
it allows all default autobuilder connections, as well as ssh and http. The
key:value pair 'network':'autobuilder-network' can then be added to your
-ProvisionGoogleVM buildstep for use.
+ProvisionGoogleVMs buildstep, allowing provisioned machines to connect to
+controllers.
Fourth, additional workers must be added to yocto-controller/controller.cfg
with the format:
BuildWorker("worker", "", max_builds=3)
where N is 0...however many workers you want and WORKER-PASSWORD is the default
-password.
+password. For example, if I wanted to add three workers for the cloud builder:
+
+c['workers'] = [BuildWorker("example-worker", "eeieHim8V", max_builds=3),]
+
+would be changed to this:
+
+c['workers'] = [BuildWorker("example-worker", "eeieHim8V", max_builds=3),
+BuildWorker("worker0", "eeieHim8V", max_builds=3),
+BuildWorker("worker1", "eeieHim8V", max_builds=3),
+BuildWorker("worker2", "eeieHim8V", max_builds=3),]
Finally, the worker-init script is only an example and has a couple
-limitations. It has only been tested for debian VMs and will likely not work
-on others. Also, the and fields will need to be manually
-edited in it if your autobuilder was created before the addition of the
-GoogleVM buildsteps or if the controller machine is behind a firewall.
+limitations itself. It has only been tested for debian VMs and will likely not
+work on others. Also, the and fields will need to be
+manually edited in it if your autobuilder was created before the addition of
+the GoogleVM buildsteps or if the controller machine is behind a firewall.
HowTo
--
@@ -41,14 +51,18 @@ HowTo
Assuming that the limitations above have been dealt with, this will provision
one new VM and completely set it up, including connections
-{'ProvisionGoogleVM':{
-'vmname':'autobuilder-worker-startup-test',
-'vmcount':1,
-'zone':'us-central1-a',
-'machine':'n1-standard-4',
-'disksize':'230GB',
-'startupscript':'/home/autobuilder/yocto-autobuilder/bin/worker-init',
-'network':'autobuilder-network'}}
+{'ProvisionGoogleVMs':{
+'vmname':'autobuilder-worker-startup-test',
+'vmcount':1,
+'zone':'us-central1-a',
+'machine':'n1-standard-4',
+'disksize':'230GB',
+'startupscript':'/path/to/yocto-autobuilder/bin/worker-init',
+'network':'autobuilder-network'}}
+
+And this will delete the entire VM when you are finished with it:
+{'DeleteGoogleVMs': {}}
-{'DeleteGoogleVM': {}}
+Note that the current implementation of DeleteGoogleVM requires that its in the
+same buildset as CreateGoogleVM.
--
1.9.1
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
[yocto] [PATCH 10/13] DeleteGoogleVM.py: Rename to DeleteGoogleVMs.py
It can delete multiple at once if multiple were created at once.
Signed-off-by: California Sullivan
---
.../autobuilder/buildsteps/DeleteGoogleVM.py | 52 --
.../autobuilder/buildsteps/DeleteGoogleVMs.py | 52 ++
2 files changed, 52 insertions(+), 52 deletions(-)
delete mode 100644
lib/python2.7/site-packages/autobuilder/buildsteps/DeleteGoogleVM.py
create mode 100644
lib/python2.7/site-packages/autobuilder/buildsteps/DeleteGoogleVMs.py
diff --git
a/lib/python2.7/site-packages/autobuilder/buildsteps/DeleteGoogleVM.py
b/lib/python2.7/site-packages/autobuilder/buildsteps/DeleteGoogleVM.py
deleted file mode 100644
index 29b45a4..000
--- a/lib/python2.7/site-packages/autobuilder/buildsteps/DeleteGoogleVM.py
+++ /dev/null
@@ -1,52 +0,0 @@
-'''
-Created on Aug 13, 2014
-
-__author__ = "California Sullivan"
-__copyright__ = "Copyright 2014, Intel Corp."
-__credits__ = ["California Sullivan"]
-__license__ = "GPL"
-__version__ = "2.0"
-__maintainer__ = "Elizabeth Flanagan"
-__email__ = "elizabeth.flana...@intel.com"
-'''
-
-from buildbot.steps.shell import ShellCommand
-
-class DeleteGoogleVM(ShellCommand):
-haltOnFailure = True
-flunkOnFailure = True
-name = "Delete Google VM"
-def __init__(self, factory, argdict=None, **kwargs):
-
-self.factory = factory
-self.description = "Deleting Google VMs"
-for k, v in argdict.iteritems():
-if type(v) is bool:
-setattr(self, k, str(v))
-else:
-setattr(self, k, v)
-# Timeout needs to be passed to LoggingBuildStep as a kwarg
-self.timeout = 1000
-kwargs['timeout']=self.timeout
-ShellCommand.__init__(self, **kwargs)
-
-
-def start(self):
-self.vmnames = self.getProperty("vmnames")
-self.vmcount = self.getProperty("vmcount")
-self.zone = self.getProperty("zone")
-
-if self.vmnames is not None and self.vmnames != "":
-self.command = "gcloud compute instances delete"
-self.command += " " + self.vmnames
-
-if self.zone is not None and self.zone in ["us-central1-a",
"us-central1-b", "us-central1-f", "europe-west1-a", "europe-west1-b",
"asia-east1-a", "asia-east1-b", "asia-east1-b"]:
-self.command += " --zone " + self.zone
-else:
-self.command += " --zone us-central1-a"
-self.command += " --quiet"
-ShellCommand.start(self)
-
-def describe(self, done=False):
-description = ShellCommand.describe(self,done)
-return description
diff --git
a/lib/python2.7/site-packages/autobuilder/buildsteps/DeleteGoogleVMs.py
b/lib/python2.7/site-packages/autobuilder/buildsteps/DeleteGoogleVMs.py
new file mode 100644
index 000..b0071cf
--- /dev/null
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/DeleteGoogleVMs.py
@@ -0,0 +1,52 @@
+'''
+Created on Aug 13, 2014
+
+__author__ = "California Sullivan"
+__copyright__ = "Copyright 2014, Intel Corp."
+__credits__ = ["California Sullivan"]
+__license__ = "GPL"
+__version__ = "2.0"
+__maintainer__ = "Elizabeth Flanagan"
+__email__ = "elizabeth.flana...@intel.com"
+'''
+
+from buildbot.steps.shell import ShellCommand
+
+class DeleteGoogleVMs(ShellCommand):
+haltOnFailure = True
+flunkOnFailure = True
+name = "Delete Google VMs"
+def __init__(self, factory, argdict=None, **kwargs):
+
+self.factory = factory
+self.description = "Deleting Google VMs"
+for k, v in argdict.iteritems():
+if type(v) is bool:
+setattr(self, k, str(v))
+else:
+setattr(self, k, v)
+# Timeout needs to be passed to LoggingBuildStep as a kwarg
+self.timeout = 1000
+kwargs['timeout']=self.timeout
+ShellCommand.__init__(self, **kwargs)
+
+
+def start(self):
+self.vmnames = self.getProperty("vmnames")
+self.vmcount = self.getProperty("vmcount")
+self.zone = self.getProperty("zone")
+
+if self.vmnames is not None and self.vmnames != "":
+self.command = "gcloud compute instances delete"
+self.command += " " + self.vmnames
+
+if self.zone is not None and self.zone in ["us-central1-a",
"us-central1-b", "us-central1-f", "europe-west1-a", "europe-west1-b",
"asia-east1-a", "asia-east1-b", "asia-east1-b"]:
+self.command += " --zone " + self.zone
+else:
+self.command += " --zone us-central1-a"
+self.command += " --quiet"
+ShellCommand.start(self)
+
+def describe(self, done=False):
+description = ShellCommand.describe(self,done)
+return description
--
1.9.1
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
[yocto] [PATCH 13/13] ProvisionGoogleVMs.py: Delete VM before trying to provision it.
In the case that somebody turned off a build before it completed, the buildstep would attempt and fail to procure a new VM, failing the build. Signed-off-by: California Sullivan --- .../autobuilder/buildsteps/ProvisionGoogleVMs.py | 16 ++-- 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/ProvisionGoogleVMs.py b/lib/python2.7/site-packages/autobuilder/buildsteps/ProvisionGoogleVMs.py index d0bc6eb..262526f 100644 --- a/lib/python2.7/site-packages/autobuilder/buildsteps/ProvisionGoogleVMs.py +++ b/lib/python2.7/site-packages/autobuilder/buildsteps/ProvisionGoogleVMs.py @@ -19,6 +19,7 @@ class ProvisionGoogleVMs(ShellCommand): name = "Provision Google VMs" def __init__(self, factory, argdict=None, **kwargs): self.vmname=None +self.vmnames="" self.vmcount=1 self.startupscript=None self.metadata=None @@ -38,15 +39,18 @@ class ProvisionGoogleVMs(ShellCommand): if self.vmname is None or self.vmname == "": self.finished(FAILURE) else: -self.command = "gcloud compute instances create" -self.vmnames = "" +self.command = "" +if self.zone is None or self.zone not in ["us-central1-a", "us-central1-b", "us-central1-f", "europe-west1-a", "europe-west1-b", "asia-east1-a", "asia-east1-b", "asia-east1-b"]: +self.zone = "us-central1-a" for x in range(0, self.vmcount): self.vmnames += " " + self.vmname + "-" + str(x) +self.command += " gcloud compute instances delete" self.command += self.vmnames -if self.zone is not None and self.zone in ["us-central1-a", "us-central1-b", "us-central1-f", "europe-west1-a", "europe-west1-b", "asia-east1-a", "asia-east1-b", "asia-east1-b"]: -self.command += " --zone " + self.zone -else: -self.command += " --zone us-central1-a" +self.command += " --zone " + self.zone + " --quiet;" +self.command += " gcloud compute instances create" +self.command += self.vmnames +self.command += " --zone " + self.zone + if self.disksize is not None and self.disksize != "": self.command += " --boot-disk-size " + self.disksize else: -- 1.9.1 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [PATCH 01/13] buildsteps: Add ProvisionGoogleVM.py
See https://bugzilla.yoctoproject.org/show_bug.cgi?id=5975
On Tue, Sep 23, 2014 at 10:13 AM, wrote:
> From: California Sullivan
>
> This build step provisions a Google VM using the gcloud command line tool.
>
> Signed-off-by: California Sullivan
> ---
> .../autobuilder/buildsteps/ProvisionGoogleVM.py| 85
> ++
> 1 file changed, 85 insertions(+)
> create mode 100644
> lib/python2.7/site-packages/autobuilder/buildsteps/ProvisionGoogleVM.py
>
> diff --git
> a/lib/python2.7/site-packages/autobuilder/buildsteps/ProvisionGoogleVM.py
> b/lib/python2.7/site-packages/autobuilder/buildsteps/ProvisionGoogleVM.py
> new file mode 100644
> index 000..fb667dd
> --- /dev/null
> +++
> b/lib/python2.7/site-packages/autobuilder/buildsteps/ProvisionGoogleVM.py
> @@ -0,0 +1,85 @@
> +'''
> +Created on Aug 13, 2014
> +
> +__author__ = "California Sullivan"
> +__copyright__ = "Copyright 2014, Intel Corp."
> +__credits__ = ["California Sullivan"]
> +__license__ = "GPL"
> +__version__ = "2.0"
> +__maintainer__ = "Elizabeth Flanagan"
> +__email__ = "elizabeth.flana...@intel.com"
> +'''
> +
> +from buildbot.steps.shell import ShellCommand
> +
> +class ProvisionGoogleVM(ShellCommand):
> +haltOnFailure = True
> +flunkOnFailure = True
> +name = "Provision Google VM"
> +def __init__(self, factory, argdict=None, **kwargs):
> +self.vmname=None
> +self.vmcount=1
> +self.startupscript=None
> +self.metadata=None
> +self.zone="us-central1-a"
> +self.image=None
> +self.machine=None
> +self.network=None
> +self.disksize=None
> +self.factory = factory
> +self.description = "Provision Google VM"
> +for k, v in argdict.iteritems():
> +if type(v) is bool:
> +setattr(self, k, str(v))
> +else:
> +setattr(self, k, v)
> +
> +if self.vmname is None or self.vmname == "":
> +self.finished(FAILURE)
> +else:
> +self.command = "gcloud compute instances create"
> +self.vmnames = ""
> +for x in range(0, self.vmcount):
> +self.vmnames += " " + self.vmname + "-" + str(x)
> +self.command += self.vmnames
> +if self.zone is not None and self.zone in ["us-central1-a",
> "us-central1-b", "us-central1-f", "europe-west1-a", "europe-west1-b",
> "asia-east1-a", "asia-east1-b", "asia-east1-b"]:
> +self.command += " --zone " + self.zone
> +else:
> +self.command += " --zone us-central1-a"
> +if self.disksize is not None and self.disksize != "":
> +self.command += " --boot-disk-size " + self.disksize
> +else:
> +self.command += " --boot-disk-size 200GB"
> +if self.image is None:
> +self.command += " --image debian-7"
> +else:
> +self.command += " --image " + self.image
> +
> +if self.machine is not None and self.machine in ["g1-small",
> "f1-micro", "n1-standard-1", "n1-standard-2", "n1-standard-4",
> "n1-standard-8", "n1-standard-16", "n1-highmem-2", "n1-highmem-4"]:
> +self.command += " --machine-type " + self.machine
> +else:
> +self.command += " --machine-type n1-standard-1"
> +if self.network is not None and self.network != "":
> +self.command += " --network " + self.network
> +if self.startupscript is not None and self.startupscript !=
> "":
> +self.command += " --metadata-from-file startup-script=" +
> self.startupscript
> +if self.metadata is not None and self.metadata != "":
> +self.command += " --metadata " + self.scriptargs
> +self.command += " 1> /dev/null";
> +
> +# Timeout needs to be passed to LoggingBuildStep as a kwarg
> +self.timeout = 1000
> +kwargs['timeout']=self.timeout
> +ShellCommand.__init__(self, **kwargs)
> +
> +def describe(self, done=False):
> +description = ShellCommand.describe(self,done)
> +return description
> +
> +def commandComplete(self, cmd):
> +if not cmd.didFail():
> +self.setProperty("vmnames", self.vmnames)
> +self.setProperty("zone", self.zone)
> +self.setProperty("vmcount", self.vmcount)
> +
> +
> --
> 1.9.1
>
> --
> ___
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [AUTOBUILDER PATCH] Initial commit of Yocto Autobuilder Dev Docs
Signed-off-by: California Sullivan On Mon, Sep 22, 2014 at 3:23 PM, Elizabeth Flanagan < elizabeth.flana...@intel.com> wrote: > The first pass of the yocto-autobuilder developer document is available at: > > > http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder/log/?h=eflanagan/docs > > The document still has some TODOs and missing spots, but at 23 pages long > it's > reached the point where the community should find it somewhat useful. > > Development is occuring for the time being on Google Docs at: > > > https://docs.google.com/document/d/1VeShmhDnqN27HXMX5emgmHFoK8Cu9SzVYilmGjtMTtI > > Elizabeth Flanagan (1): > Initial Commit of Documentation > > docs/YoctoAutobuilderDevelopersDocument.html | 1 + > docs/images/image00.png | Bin 0 -> 67622 bytes > 2 files changed, 1 insertion(+) > create mode 100644 docs/YoctoAutobuilderDevelopersDocument.html > create mode 100644 docs/images/image00.png > > -- > 1.9.0 > > -- > ___ > yocto mailing list > yocto@yoctoproject.org > https://lists.yoctoproject.org/listinfo/yocto > -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [PATCH] Autobuilder.py: Fix triggered build bug
I apologize for the late response - I was working through some other issues
that I wanted to make sure were unrelated to this.
I have tested this on a build that triggers a build on a remote worker, as
well as a build that triggers a build on a local worker. Both appear to
have full functionality.
---
Cal
On Tue, Sep 2, 2014 at 1:45 PM, Flanagan, Elizabeth <
elizabeth.flana...@intel.com> wrote:
> On Thu, Aug 28, 2014 at 4:33 PM, California Sullivan
> wrote:
> > When a build was configured to use the TriggerBuilds build step, it would
> > take on the buildslave of the last triggered build rather than its own.
> >
> > Signed-off-by: California Sullivan
> > ---
> > lib/python2.7/site-packages/autobuilder/Autobuilder.py | 3 +--
> > 1 file changed, 1 insertion(+), 2 deletions(-)
> >
> > diff --git a/lib/python2.7/site-packages/autobuilder/Autobuilder.py
> > b/lib/python2.7/site-packages/autobuilder/Autobuilder.py
> > index a582a03..2e2e899 100644
> > --- a/lib/python2.7/site-packages/autobuilder/Autobuilder.py
> > +++ b/lib/python2.7/site-packages/autobuilder/Autobuilder.py
> > @@ -58,7 +58,7 @@ class Autobuilder:
> > self.set_props = {}
> > self.repos = []
> > self.properties = []
> > -self.builders = None
> > +self.builders = self.parseBuilders(buildset)
> > self.schedprops.append(NestedParameter(name='ss_' +
> buildset,
> > label="" + buildset +
> "
> > defaults:", fields=[FixedParameter(name="dummy",
> > default="dummy")]))
> > self.parseBuildSet(buildset)
> > locals()['buildset_%s' %
> > buildset]=BuildSet.BuildSet(name=buildset,
> > @@ -86,7 +86,6 @@ class Autobuilder:
> >
> > def parseBuildSet(self, buildset):
> > if buildset is not "BuildSets":
> > -self.builders=self.parseBuilders(buildset)
> > self.parseProps(buildset)
> > self.parseRepos(buildset)
> > self.parseSteps(buildset)
> > --
> > 1.9.1
> >
> >
> > --
> > ___
> > yocto mailing list
> > yocto@yoctoproject.org
> > https://lists.yoctoproject.org/listinfo/yocto
> >
>
>
> This looks ok, but I'd like to test it a bit before I shove it into
> production. What have you tested this with (example: I tested it on a
> single build triggering another one, or a build that triggers
> itself...) With Trigger being a bit of core functionality, it would
> help a bit to know this.
>
> -b
>
>
> --
> Elizabeth Flanagan
> Yocto Project
> Build and Release
>
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
[yocto] [PATCH] Autobuilder.py: Fix triggered build bug
When a build was configured to use the TriggerBuilds build step, it would
take on the buildslave of the last triggered build rather than its own.
Signed-off-by: California Sullivan
---
lib/python2.7/site-packages/autobuilder/Autobuilder.py | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/lib/python2.7/site-packages/autobuilder/Autobuilder.py
b/lib/python2.7/site-packages/autobuilder/Autobuilder.py
index a582a03..2e2e899 100644
--- a/lib/python2.7/site-packages/autobuilder/Autobuilder.py
+++ b/lib/python2.7/site-packages/autobuilder/Autobuilder.py
@@ -58,7 +58,7 @@ class Autobuilder:
self.set_props = {}
self.repos = []
self.properties = []
-self.builders = None
+self.builders = self.parseBuilders(buildset)
self.schedprops.append(NestedParameter(name='ss_' + buildset,
label="" + buildset + "
defaults:", fields=[FixedParameter(name="dummy",
default="dummy")]))
self.parseBuildSet(buildset)
locals()['buildset_%s' %
buildset]=BuildSet.BuildSet(name=buildset,
@@ -86,7 +86,6 @@ class Autobuilder:
def parseBuildSet(self, buildset):
if buildset is not "BuildSets":
-self.builders=self.parseBuilders(buildset)
self.parseProps(buildset)
self.parseRepos(buildset)
self.parseSteps(buildset)
--
1.9.1
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
