Hmm, that would be cool if we don't need Selenium to report results.
Keeping Selenium synced up is a PITA.
Regarding declarative languages, the reason I chose MXML when writing
Mustella was that, in Flash, you simply have to stop running AS code in
order for the player to render the screen and determine the final size of
things. I've never looked to see how FlexUnit handles this, but I'm not
clear how a test can be written in AS as:
@Test
Function MyTest() {
SetAProperty();
AssertSomeOtherProperty();
}
unless there are mocks for the player, or you don't run any tests that
depend on the player.
In Mustella, the same test looks like:
<TestCase id="MyTest">
<SetProperty ... />
<AssertPropertyValue .../>
</TestCase>
And then on Flash, the Mustella engine sets the property and waits for the
player to refresh before running AssertProperty. And in the browser, the
Mustella engine doesn't have to wait. But at Adobe, the "Unit" was a
component, not APIs or methods within a component, and thus we needed the
player to do its thing in most cases. FlexUnit is working against smaller
units and Mustella is really more of a integration testing engine.
My other point was that, since the set of available test steps is
well-defined, you can't write a test with infinite loops, or screw up the
setting of a property, etc.
HTH,
-Alex
On 11/4/17, 10:46 PM, "Harbs" <[email protected]> wrote:
>Josh got his tests to be reported in the terminal console and he added
>integration with Travis. Integration of node processes is pretty standard
>in the JS world. I have not yet looked into integration with ant (or
>Maven), but I can’t imagine that running node via ant is much different
>than any other process.
>
>> The advantage of declarative testing languages is that you get to
>> constrain the kinds of things that a test can do, and abstract when the
>> test harness does it. It is much harder to interrupt sequential lines
>>of
>> ActionScript/JavaScript.
>
>I did not get this at all. Do you have any examples or links which show
>what you mean?
>
>I’m somewhat embarrassed to say that I’ve never really used much
>automated testing, so a lot of these concepts are more theoretical to me
>than they really should be… ;-)
>
>Harbs
>
>> On Nov 5, 2017, at 8:17 AM, Alex Harui <[email protected]> wrote:
>>
>> Those warnings do not seem to affect the success of the tests on SWF for
>> me.
>>
>> IMO, the big challenge is in capturing test results so it can be
>>reported
>> in a CI dashboard. Having a GUI display results in Flash or in a
>>browser
>> is a great first step, but I believe going from there to getting
>>Selenium
>> to report individual test failures may be trickier.
>>
>> Peter tried to write up Mustella a long time ago. It might need
>>updating.
>> See:
>>
>>https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.ap
>>ache.org%2Fconfluence%2Fdisplay%2FFLEX%2FMustella%2BOverview&data=02%7C01
>>%7C%7C0414ee815f254be8230f08d524190130%7Cfa7b1b5a7b34438794aed2c178decee1
>>%7C0%7C0%7C636454612170971092&sdata=70%2BMRX6UcjeFP%2BB43zSZDo53bgq0oxOZT
>>4G6uLtYzII%3D&reserved=0
>><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.a
>>pache.org%2Fconfluence%2Fdisplay%2FFLEX%2FMustella%2BOverview&data=02%7C0
>>1%7C%7C0414ee815f254be8230f08d524190130%7Cfa7b1b5a7b34438794aed2c178decee
>>1%7C0%7C0%7C636454612170971092&sdata=70%2BMRX6UcjeFP%2BB43zSZDo53bgq0oxOZ
>>T4G6uLtYzII%3D&reserved=0>
>>
>> The advantage of declarative testing languages is that you get to
>> constrain the kinds of things that a test can do, and abstract when the
>> test harness does it. It is much harder to interrupt sequential lines
>>of
>> ActionScript/JavaScript.
>>
>> HTH,
>> -Alex
>>
>> On 11/4/17, 3:13 PM, "Harbs" <[email protected]
>><mailto:[email protected]>> wrote:
>>
>>> It looks like FlexUnit is currently broken.
>>>
>>> I’m getting the following output when trying to run the FlexUnit tests
>>>on
>>> Core and Basic:
>>>
>>> [mxmlc]
>>>
>>>/Users/harbs/Documents/ApacheRoyale/flex-flexunit/FlexUnit4/target/flexu
>>>ni
>>> t-4.3.0-20140410-as3_4.12.0.swc Warning: The definition
>>> mx.utils.ObjectUtil depended on by org.flexunit.runner.Description in
>>>the
>>> SWC
>>>
>>>/Users/harbs/Documents/ApacheRoyale/flex-flexunit/FlexUnit4/target/flexu
>>>ni
>>> t-4.3.0-20140410-as3_4.12.0.swc could not be found
>>> [mxmlc]
>>> [mxmlc]
>>> [mxmlc]
>>>
>>>/Users/harbs/Documents/ApacheRoyale/flex-flexunit/FlexUnit4/target/flexu
>>>ni
>>> t-4.3.0-20140410-as3_4.12.0.swc Warning: The definition
>>> mx.utils.StringUtil depended on by flexunit.framework.AsyncTestHelper
>>>in
>>> the SWC
>>>
>>>/Users/harbs/Documents/ApacheRoyale/flex-flexunit/FlexUnit4/target/flexu
>>>ni
>>> t-4.3.0-20140410-as3_4.12.0.swc could not be found
>>>
>>> Has anyone looked at this yet?
>>>
>>> Harbs
>>>
>>>> On Nov 4, 2017, at 10:45 PM, Harbs <[email protected]> wrote:
>>>>
>>>> Thanks! That looks very useful.
>>>>
>>>> I started work on a feature/testing branch. I’ll try to merge what you
>>>> did into what Josh did. I’m going to try to get the existing Core
>>>>and/or
>>>> Basic tests working in both swf and node js output tomorrow. We’ll see
>>>> how well I do… ;-)
>>>>
>>>> We might need different configs for different testing environments,
>>>>but
>>>> I’ll see if we can keep the divergence of the environs is minimal as
>>>> possible.
>>>>
>>>> Once I get the basics worked out, I’ll likely have an idea what others
>>>> can help with. Let’s see if we can whip the Royale tests into shape.
>>>>:-)
>>>>
>>>> Harbs
>>>>
>>>>> On Nov 4, 2017, at 10:30 PM, Greg Dove <[email protected]> wrote:
>>>>>
>>>>> Harbs, I don't know if it still works since changes to Royale, but I
>>>>> had
>>>>> something rudimentary for cross-target unit testing quite shortly
>>>>>after
>>>>> working on the reflection support last year, in fact that was my
>>>>> primary
>>>>> reason for choosing to focus on reflection at the time.
>>>>>
>>>>> It was visual output only (i.e. just to look at the test results
>>>>> output in
>>>>> the browser) and the goal was to get some compatibility with flexunit
>>>>> (so
>>>>> that the same swf-based flexunit tests that were running in the build
>>>>> -e..g
>>>>> BinaryData tests- could be run visually side by side between swf and
>>>>> js).
>>>>> I also added a new TestVariance metadata to account for known
>>>>> (expected?)
>>>>> variation between swf and js. This was needed to cover (for example)
>>>>> things
>>>>> like testing Reflection classes themselves, where the results can be
>>>>> different between the targets based on the 'native' base classes
>>>>>(e..g
>>>>> EventDispatcher inheritance chain).
>>>>>
>>>>> Some of that code might also be useful to harvest for what you are
>>>>> doing,
>>>>> I'm not sure...
>>>>>
>>>>> It's here:
>>>>>
>>>>>
>>>>>https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithu
>>>>>b
>>>>><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
>>>>>ub>.
>>>>>
>>>>>com%2Fapache%2Froyale-asjs%2Ftree%2Fdevelop%2Fmanualtests%2FUnitTests&
>>>>>da
>>>>>
>>>>>ta=02%7C01%7C%7C67a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794a
>>>>>ed
>>>>>
>>>>>2c178decee1%7C0%7C0%7C636454304479378270&sdata=DX9gXiLqUoFQizp6xrakUwx
>>>>>jl
>>>>> cdyHMlZO5mOzDSiKDQ%3D&reserved=0
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Nov 5, 2017 at 7:30 AM, Harbs <[email protected]
>>>>><mailto:[email protected]>> wrote:
>>>>>
>>>>>> Awesome!
>>>>>>
>>>>>> I assume that this means I can code the code to the Royale repo
>>>>>>under
>>>>>> Apache 2. Right?
>>>>>>
>>>>>> Harbs
>>>>>>
>>>>>>> On Nov 3, 2017, at 5:37 PM, Josh Tynjala <[email protected]
>>>>>>><mailto:[email protected]>>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Please feel free to use my test runner. I'm happy to commit it
>>>>>>> somewhere
>>>>>> to make it official Apache code, if necessary.
>>>>>>>
>>>>>>> - Josh
>>>>>>>
>>>>>>> On 2017-11-03 04:19, Harbs <[email protected]
>>>>>>><mailto:[email protected]>> wrote:
>>>>>>>> One topic which keeps coming up is better test coverage for
>>>>>>>>Royale.
>>>>>>>>
>>>>>>>> I think this is becoming a critical issue for a few reasons:
>>>>>>>> 1. As we get close to version 1.0 it’s necessary to have good
>>>>>>>>test
>>>>>> coverage for confidence of quality and confidence that we don’t
>>>>>> introduce
>>>>>> recessive bugs.
>>>>>>>> 2. It’s really hard to accept Github pull requests without
>>>>>>>> examining
>>>>>> the code VERY well that it does not introduce recessive bugs. CI
>>>>>> which runs
>>>>>> automated tests could give a preliminary test on pull requests to
>>>>>> ensure
>>>>>> that they don’t break anything. If the pull requests do break
>>>>>> something,
>>>>>> it allows the requester to fix the problem with confidence without
>>>>>> taking
>>>>>> others’ time.
>>>>>>>>
>>>>>>>> I think we need to break up testing into pieces and figure out a
>>>>>> strategy to implement automated tests in a way that they are
>>>>>> maintainable.
>>>>>>>>
>>>>>>>> Some points:
>>>>>>>> 1. I think integration into something like Travis would be very
>>>>>>>> helpful.
>>>>>>>> 2. I think there’s a Jenkins plugin for building pull requests.
>>>>>>>> Not
>>>>>> sure how useful it is.[1]
>>>>>>>> 3. Josh has created a Node.js-compatible test-runner architecture
>>>>>>>> which
>>>>>> could be useful for unit tests on parts of the framework which
>>>>>> don’t rely
>>>>>> on browser features. (i.e. models and the like) He mentioned the
>>>>>> possibility of donating it, and I think that would be a very useful
>>>>>> feature.
>>>>>>>> 4. For UI and integration tests, there seem to be some pretty cool
>>>>>> integrations using Selenium.[2][3]
>>>>>>>> 5. I think the main testing effort needs to be using JS and
>>>>>>>> something
>>>>>> like Josh’s testing framework for non-UI pieces and some
>>>>>>easy-to-use
>>>>>> Selenium integration for visual pieces and integration tests.
>>>>>>>> 6. We probably also want some API endpoints we can test off of for
>>>>>> integration testing.
>>>>>>>>
>>>>>>>> I’m willing to invest time into this.
>>>>>>>>
>>>>>>>> It’s going to be a lot of work building out the tests and I
>>>>>>>>think
>>>>>>>> we
>>>>>> need a plan for that. My thoughts:
>>>>>>>>
>>>>>>>> 1. Step one is to make it easy to write meaningful automated tests
>>>>>>>> and
>>>>>> establish a clear pattern.
>>>>>>>> 2. Step two is to start writing tests starting from the
>>>>>> most-used/easiest to beak pieces and work out from there.
>>>>>>>> 3. Once the pattern is established, any new pieces MUST have
>>>>>>>>testing
>>>>>> coverage.
>>>>>>>> 4. When fixing bugs, attention should be paid to adding testing
>>>>>>>>for
>>>>>> that component.
>>>>>>>> 5. When a pull request comes in on a piece which does not have
>>>>>>>>unit
>>>>>> test, a test must be written before accepting the pull request. The
>>>>>> test
>>>>>> does not need to be written by the requester. Before examining the
>>>>>> request,
>>>>>> the test should be written to pass for expected behavior and fail
>>>>>>for
>>>>>> the
>>>>>> bug that the pull request is attempting to fix (assuming the pull
>>>>>> request
>>>>>> is to fix a bug).
>>>>>>>>
>>>>>>>> Thoughts?
>>>>>>>> Harbs
>>>>>>>>
>>>>>>>> P.S. I’m thinking of coming to the US in late December/early
>>>>>>>> January.
>>>>>> I would be interested in getting together for a hacking session with
>>>>>> folks
>>>>>> who are available.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>[1]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2
>>>>>>>>Fw
>>>>>>>><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fw
>>>>>>>>>
>>>>>>>>
>>>>>>>>iki.jenkins.io%2Fdisplay%2FJENKINS%2FGitHub%2Bpull&data=02%7C01%7C%
>>>>>>>>7C
>>>>>>>>
>>>>>>>>67a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178decee1
>>>>>>>>%7
>>>>>>>>
>>>>>>>>C0%7C0%7C636454304479378270&sdata=cnyBtnVycpg3Aa7Hfel3dkAlez2m7M0uS
>>>>>>>>l3
>>>>>>>> f0ezbSZY%3D&reserved=0+
>>>>>> request+builder+plugin
>>>>>>
>>>>>><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwik
>>>>>>i
>>>>>><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwik
>>>>>>i>.
>>>>>>
>>>>>>jenkins.io%2F&data=02%7C01%7C%7C67a58a332b6942a2022108d523d15d35%7Cfa
>>>>>>7b
>>>>>>
>>>>>>1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454304479378270&sdata=eqe6e
>>>>>>c%
>>>>>> 2F%2FIatSxdUuHU0Lo8o3WxySBkdSvUNa6i9NQ%2FI%3D&reserved=0
>>>>>> display/JENKINS/GitHub+pull+request+builder+plugin>
>>>>>>>>
>>>>>>>>
>>>>>>>>[2]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2
>>>>>>>>Fd
>>>>>>>><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fd
>>>>>>>>>
>>>>>>>> ocs.travis-ci.com
>>>>>>>><https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Foc
>>>>>>>>s.travis-ci.com%2F&data=02%7C01%7C%7C0414ee815f254be8230f08d5241901
>>>>>>>>30%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454612170971092&
>>>>>>>>sdata=pnhMpAYNngULtMbB%2BHh43o8kTA5%2Bxu9sOciWDDW3dLM%3D&reserved=0
>>>>>>>>>%2Fuser%2Fgui-and-headless-browsers%2F&data=02%7C01%
>>>>>>>>
>>>>>>>>7C%7C67a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178d
>>>>>>>>ec
>>>>>>>>
>>>>>>>>ee1%7C0%7C0%7C636454304479378270&sdata=zK%2FdOBmWJUsF7SylIQtMQQpAeO
>>>>>>>>hw
>>>>>>>> 03sjOytutB4rGCY%3D&reserved=0 <
>>>>>>
>>>>>>
>>>>>>https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs
>>>>>>.t
>>>>>><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdoc
>>>>>>s.t>
>>>>>> ravis-ci.com
>>>>>><https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fravi
>>>>>>s-ci.com%2F&data=02%7C01%7C%7C0414ee815f254be8230f08d524190130%7Cfa7b
>>>>>>1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454612170971092&sdata=jOUFN
>>>>>>%2FhuadJ3C%2B7U6xfEvhq5EZpT9OfFWaMCAkfhg%2FY%3D&reserved=0>%2Fuser%2F
>>>>>>gui-and-headless-browsers%2F&data=02%7C01%7C%7C67
>>>>>>
>>>>>>a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178decee1%7C0
>>>>>>%7
>>>>>>
>>>>>>C0%7C636454304479378270&sdata=zK%2FdOBmWJUsF7SylIQtMQQpAeOhw03sjOytut
>>>>>>B4
>>>>>> rGCY%3D&reserved=0>
>>>>>>>>
>>>>>>>>
>>>>>>>>[3]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2
>>>>>>>>Fs
>>>>>>>><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fs
>>>>>>>>>
>>>>>>>> aucelabs.com
>>>>>>>><https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fau
>>>>>>>>celabs.com%2F&data=02%7C01%7C%7C0414ee815f254be8230f08d524190130%7C
>>>>>>>>fa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454612170971092&sdata
>>>>>>>>=yfrl9UGpLIpcUZRLUqyAiPxp64tMr5ow7aC8ZqhQFOA%3D&reserved=0>%2Fprodu
>>>>>>>>cts%2Fintegrations&data=02%7C01%7C%7C67a58a332b69
>>>>>>>>
>>>>>>>>42a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C6
>>>>>>>>36
>>>>>>>>
>>>>>>>>454304479378270&sdata=kwfto9xiP0ygeBrEqGz9ucdC%2FNXSRVOQho9qUW1mqdw
>>>>>>>>%3
>>>>>>>> D&reserved=0
>>>>>>>>
>>>>>>>><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fs
>>>>>>>>au
>>>>>>>><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fs
>>>>>>>>au>
>>>>>>>> celabs.com
>>>>>>>><https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fce
>>>>>>>>labs.com%2F&data=02%7C01%7C%7C0414ee815f254be8230f08d524190130%7Cfa
>>>>>>>>7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454612170971092&sdata=A
>>>>>>>>Nmmle6KQFo99Z3l8ie1q9xIe4SJOMQK%2FgCQbYipRwo%3D&reserved=0>%2F&data
>>>>>>>>=02%7C01%7C%7C67a58a332b6942a2022108d523d15d35%7Cfa
>>>>>>>>
>>>>>>>>7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454304479378270&sdata=x
>>>>>>>>pG
>>>>>>>> G85mRtcFwwG7nbRfT7oLMPm4VPeHxe4trkG1wrAw%3D&reserved=0
>>>>>> products/integrations>
>
