Title: [101662] trunk/Tools
- Revision
- 101662
- Author
- aro...@apple.com
- Date
- 2011-12-01 05:36:39 -0800 (Thu, 01 Dec 2011)
Log Message
Use curl for downloading builds to test slaves
Buildbot's built-in file transfer capabilities are transferring files much more slowly than
curl does. This seems to be due to a combination of buildbot being single-threaded (and thus
often busy servicing web requests, etc.) and using an IPC mechanism with fairly high
overhead (courtesy of the Twisted framework).
Eventually we'd like to make Buildbot's built-in transferring more efficient. In the
meantime we'll try using curl for downloads and keep thinking about a solution for uploads.
This should make the test slaves quite a bit faster, and should ease the CPU load on
build.webkit.org a bit.
Part of <http://webkit.org/b/73484> Mac slaves take 7+ minutes to upload or download a build
Reviewed by Mark Rowe.
* BuildSlaveSupport/build.webkit.org-config/master.cfg: Added the CreateWebKitBuildDirectory
to Test slaves. It's run just before DownloadBuiltProduct.
(DownloadBuiltProduct): Changed to be a normal ShellCommand that invokes curl to perform the
download.
(CreateWebKitBuildDirectory): Added. Creates the WebKitBuild directory on a slave.
FileDownload used to do this for us automatically.
Modified Paths
Diff
Modified: trunk/Tools/BuildSlaveSupport/build.webkit.org-config/master.cfg (101661 => 101662)
--- trunk/Tools/BuildSlaveSupport/build.webkit.org-config/master.cfg 2011-12-01 13:33:53 UTC (rev 101661)
+++ trunk/Tools/BuildSlaveSupport/build.webkit.org-config/master.cfg 2011-12-01 13:36:39 UTC (rev 101662)
@@ -168,16 +168,24 @@
return result
-class DownloadBuiltProduct(transfer.FileDownload):
- slavedest = WithProperties("WebKitBuild/%(configuration)s.zip")
- mastersrc = WithProperties("archives/%(fullPlatform)s-%(architecture)s-%(configuration)s/%(got_revision)s.zip")
+class DownloadBuiltProduct(shell.ShellCommand):
+ command = ["curl", "--output", WithProperties("WebKitBuild/%(configuration)s.zip"), WithProperties(c["buildbotURL"] + "archives/%(fullPlatform)s-%(architecture)s-%(configuration)s/%(got_revision)s.zip")]
+ name = "download-built-product"
+ description = ["downloading built product"]
+ descriptionDone = ["downloaded built product"]
haltOnFailure = True
flunkOnFailure = True
- def __init__(self):
- transfer.FileDownload.__init__(self, self.mastersrc, self.slavedest)
+class CreateWebKitBuildDirectory(shell.ShellCommand):
+ command = ["mkdir", "-p", "WebKitBuild"]
+ name = "create-webkitbuild-dir"
+ description = ["creating WebKitBuild directory"]
+ descriptionDone = ["created WebKitBuild directory"]
+ haltOnFailure = True
+ flunkOnFailure = True
+
class RunJavaScriptCoreTests(shell.Test):
name = "jscore-test"
description = ["jscore-tests running"]
@@ -642,6 +650,7 @@
ExtractTestResultsClass = ExtractTestResults
def __init__(self, platform, configuration, architectures, **kwargs):
Factory.__init__(self, platform, configuration, architectures, False, **kwargs)
+ self.addStep(CreateWebKitBuildDirectory)
self.addStep(DownloadBuiltProduct)
self.addStep(ExtractBuiltProduct)
self.addStep(RunJavaScriptCoreTests, buildJSCTool=False)
Modified: trunk/Tools/ChangeLog (101661 => 101662)
--- trunk/Tools/ChangeLog 2011-12-01 13:33:53 UTC (rev 101661)
+++ trunk/Tools/ChangeLog 2011-12-01 13:36:39 UTC (rev 101662)
@@ -1,3 +1,28 @@
+2011-11-30 Adam Roben <aro...@apple.com>
+
+ Use curl for downloading builds to test slaves
+
+ Buildbot's built-in file transfer capabilities are transferring files much more slowly than
+ curl does. This seems to be due to a combination of buildbot being single-threaded (and thus
+ often busy servicing web requests, etc.) and using an IPC mechanism with fairly high
+ overhead (courtesy of the Twisted framework).
+
+ Eventually we'd like to make Buildbot's built-in transferring more efficient. In the
+ meantime we'll try using curl for downloads and keep thinking about a solution for uploads.
+ This should make the test slaves quite a bit faster, and should ease the CPU load on
+ build.webkit.org a bit.
+
+ Part of <http://webkit.org/b/73484> Mac slaves take 7+ minutes to upload or download a build
+
+ Reviewed by Mark Rowe.
+
+ * BuildSlaveSupport/build.webkit.org-config/master.cfg: Added the CreateWebKitBuildDirectory
+ to Test slaves. It's run just before DownloadBuiltProduct.
+ (DownloadBuiltProduct): Changed to be a normal ShellCommand that invokes curl to perform the
+ download.
+ (CreateWebKitBuildDirectory): Added. Creates the WebKitBuild directory on a slave.
+ FileDownload used to do this for us automatically.
+
2011-11-21 Philippe Normand <pnorm...@igalia.com>
[GTK] enable-webaudio websetting
_______________________________________________
webkit-changes mailing list
webkit-changes@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-changes
