Yes, I think it's quite complementary.
Looking forward to seeing the code land on the acceptance-test repo.
2014-05-13 4:25 GMT-07:00 Stephen Connolly <stephen.alan.conno...@gmail.com>
:
> Oh I forgot my point... I suspect that some of the acceptance tests would
> be easier to write if you use the scalability framework's DSL to set up a
> lot of the acceptance test... at least they would be less fragile as you
> have a known scenario with inter-job relationships. I have found that
> setting up complex job scenarios via Selenium can be somewhat fragile and
> you end up with brittle tests that fail at random.
>
> I do not think all acceptance tests should use the scalability framework's
> Plan model to set up the test environment (other than the bare Plan with
> plugins installed) as there is value in testing the path... but the more
> complex ones where the setup path is already covered... they can get value
>
>
> On 13 May 2014 12:21, Stephen Connolly <stephen.alan.conno...@gmail.com>wrote:
>
>> Well I have a scalability test framework that makes it very easy to test
>> jenkins at scale. I also think that the DSL for scaling jenkins is probably
>> very handy for acceptace tests. For example, here is my *current* API for
>> load testing the SSH Slaves connector:
>>
>> Plan p = new Plan();
>> Ec2MachineFactory factory = new Ec2MachineFactory(new
>> Ec2Config());
>> try (Execution e = new Execution(p)
>> .with(MasterNode.factory())
>> .with(SlaveNode.factory())
>> .machines(factory)
>> .machines(new MultitenantMachinesFactory(factory, 15),
>> PlannedSlave.class)) {
>> PlannedMaster master = p.createMaster()
>> .withPlugins("metrics", "random-job-builder")
>> .withView(new
>> PlannedCloudBeesDashboardView().withName("Dashboard"))
>> .withPluginsDisabled("translation")
>> .withExecutors(0);
>> NioSshSlaveLauncher launcher = new NioSshSlaveLauncher();
>> for (int i = 0; i < 60; i++) {
>>
>> master.withSlave(p.createSlave().withLauncher(launcher).withExecutors(2));
>> }
>> for (int f = 0; f < 10; f++) {
>> PlannedFolder folder = p.createFolder(master);
>> for (int g = 0; g < 3; g++) {
>> PlannedFolder subfolder = p.createFolder(folder);
>> for (int h = 0; h < 10; h++) {
>> p.createMockJob(subfolder);
>> }
>> }
>> }
>> e.start();
>> MasterNode executionMaster = e.resolve(master);
>> // give the user a browser window to observe the load
>>
>> Desktop.getDesktop().browse(URI.create(executionMaster.getJenkinsUrl()+"view/Dashboard/"));
>> // wait for the user to request tear-down
>> }
>>
>> I have found this *very* powerful for testing load. The above scenario
>> let me determine, for example, that on an m3.large you can have about 60
>> concurrent mock load builds using the JNLP launcher, or about 10 concurrent
>> mock load builds using the Trilead SSH launcher... above those levels by
>> more than 5 concurrent builds and Jenkins blows up. With the CloudBees
>> NioSSH launcher you can have 15 concurrent builds without slowing of build
>> times... and you can have 120 concurrent builds without Jenkins blowing up.
>>
>> What I need to do is detangle some of the CloudBees specific bits: e.g.
>> in the above example there are two classes that will not be made public:
>> PlannedCloudBeesDashboardView and NioSshSlaveLauncher because those classes
>> will decorate the provisioned Jenkins master with a CloudBees Jenkins
>> Enterprise license ;-)
>>
>> I also want to get a hashing function for a Plan so that the JUT server
>> can be made compatible in the face of requests for specific plans.
>>
>>
>> On 13 May 2014 09:59, Ulli Hafner <ullrich.haf...@gmail.com> wrote:
>>
>>> One of our students is currently trying to integrate Spock and Geb: this
>>> also requires API changes (at least we need to introduce additional
>>> interfaces for all those concrete selenium base classes). This part still
>>> is work in progress.
>>>
>>> I also thought the purpose of this project is to collect all acceptance
>>> tests for Jenkins. So if we need to change the framework we just go ahead
>>> and commit the required changes. Up to now it was not clear that we need to
>>> publish a „stable“ version at all.
>>> Are there any other plans for additional test suites that are based on
>>> the acceptance test harness project (besides your Cloudbees test suite)?
>>>
>>> Am 13.05.2014 um 02:26 schrieb Kohsuke Kawaguchi <
>>> kkawagu...@cloudbees.com>:
>>>
>>> > On 05/12/2014 04:09 PM, Stephen Connolly wrote:
>>> >> You know I have *very* strong feelings on a lot of the lower
>>> layers... but you
>>> >> also know that I haven't had the time to get the replacement layer
>>> published
>>> >> yet... It's coming... there is the possibility that it may break some
>>> stuff...
>>> >> but I am reasonably confident that it shouldn't
>>> >
>>> > Good point, so there's at least that group of changes that we should
>>> expect before we consider it solidified.
>>> >
>>> >> On 12 May 2014 23:28, Kohsuke Kawaguchi <k...@kohsuke.org <mailto:
>>> k...@kohsuke.org>>
>>> >> wrote:
>>> >>
>>> >>
>>> >> So I jumped the gun a little bit and cut a release of
>>> >> acceptance-test-harness, as it was blocking me on developing
>>> other sets of
>>> >> tests internally.
>>> >>
>>> >> But this triggered a comment from Oliver [1] in terms of what we
>>> want to see
>>> >> in the code before we consider the API stable enough.
>>> >>
>>> >> Specifically,
>>> >>
>>> >> I hoped there will be a discussion before making the API
>>> public. I
>>> >> wanted to get rid off all the public Controls exposed from
>>> page objects
>>> >> because we will not be able to preserve the API as Jenkins UI
>>> evolves.
>>> >> Changes like af98505 will surely follow.
>>> >>
>>> >>
>>> >> So first of all, I wanted to set the expectation right that 1.0
>>> and 1.1 I
>>> >> released shouldn't be considered stable.
>>> >>
>>> >>
>>> >> With that said, personally I didn't anticipate that we hold
>>> ourselves to the
>>> >> level of backward compatibility that we do for Jenkins, and
>>> public final
>>> >> Control fields are nice, namely they are very concise to write.
>>> >>
>>> >> But I'm willing to follow what we want as a whole, so if you have
>>> any strong
>>> >> feelings and thoughts on this, this is the opportunity to speak
>>> up.
>>> >>
>>> >>
>>> >> [1]
>>> >>
>>> https://github.com/jenkinsci/__acceptance-test-harness/__commit/__8a4bacb386ee0b5a34c5dd499bd74f__50b35c726d#commitcomment-__6297001
>>> >> <
>>> https://github.com/jenkinsci/acceptance-test-harness/commit/8a4bacb386ee0b5a34c5dd499bd74f50b35c726d#commitcomment-6297001
>>> >
>>> >> --
>>> >> Kohsuke Kawaguchi http://kohsuke.org/
>>> >>
>>> >> --
>>> >> You received this message because you are subscribed to the
>>> Google Groups
>>> >> "Jenkins Developers" group.
>>> >> To unsubscribe from this group and stop receiving emails from it,
>>> send an
>>> >> email to jenkinsci-dev+unsubscribe@__googlegroups.com
>>> >> <mailto:jenkinsci-dev%2bunsubscr...@googlegroups.com>.
>>> >> For more options, visit https://groups.google.com/d/__optout
>>> >> <https://groups.google.com/d/optout>.
>>> >>
>>> >>
>>> >> --
>>> >> You received this message because you are subscribed to the Google
>>> Groups
>>> >> "Jenkins Developers" group.
>>> >> To unsubscribe from this group and stop receiving emails from it,
>>> send an email
>>> >> to jenkinsci-dev+unsubscr...@googlegroups.com
>>> >> <mailto:jenkinsci-dev+unsubscr...@googlegroups.com>.
>>> >> For more options, visit https://groups.google.com/d/optout.
>>> >>
>>> >
>>> >
>>> > --
>>> > Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/
>>> > Try Jenkins Enterprise, our professional version of Jenkins
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google
>>> Groups "Jenkins Developers" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send
>>> an email to jenkinsci-dev+unsubscr...@googlegroups.com.
>>> > For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jenkinsci-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
Kohsuke Kawaguchi
--
You received this message because you are subscribed to the Google Groups
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jenkinsci-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
