Question #667037 on Sikuli changed:
https://answers.launchpad.net/sikuli/+question/667037
Status: Open => Answered
RaiMan proposed the following answer:
Though I think, that to use Python language is not too complicated and
challenging to write down basic to medium complex SikuliX based
workflows, there is always room, for making it even easier for people,
that should not be "tortured with programming".
With SikuliX workflows we have 3 challenges, to support these people:
1. a frontend, that easily allows to add the SikuliX features to a
workflow, based on clickable templates, that "know" about the already
existing workflow context and support the intuitive completion of specs
and parameters. This should be available for automation workflows and
testcases. Running such a workflow means to translate it to a runnable
script and run it.
2. a tool, that allows to capture, optimize and manage the images needed
in a workflow.
3. a recorder, that allows to setup a basic workflow while working with
some application.
I am on the way to address all 3 aspects with SikuliX2, but have to
admit, that I am some months away from a useable package.
my ideas about the 3 challenges (for all 3 there are already basic
approaches available in SikuliX2):
-- at 1: the frontend should be table-based (like the workflows with
RobotFramework).
Each line is an action like click, wait, type/write/paste.
An action can be selected from a context-menu or entered by typing in the first
column (auto-complete) and a suitable template is filled into the line.
Parameters of an action are edited by just clicking on the respective column in
the line (example: you can capture and/or manage a needed image supported by
the tool of challenge 2)
With this basic sequential workflows can be created and run easily.
Medium complex workflows additionally need repeat (loop, for, ..) and decision
(if/elif/else) features, some kind of supporting functions and import and some
features to interact with the user, while the workflow is executed.
In SikuliX2 I already have a basic approach based on a tablebased JPanel, that
translates into a runnable JavaScript script (JavaScript with Java has the
advantage, that the interpreter is JVM-builtin (Nashorn))
After having implemented such a frontend, it might be easily complemented with
a completely visual tool, that shows the workflow as a diagram with graphical
elements, that can be edited by just clicking and dragging around (you
mentioned UML).
It might be a good idea, to make this frontend browser-based.
And of course it could be implemented in Eclipse/Netbeans/IntelliJ IDEA using
their plugin frameworks or even in JEdit.
-- at 2: the tool must allow to capture images and optimize them for the use of
SikuliX features (concentrate on the key visual aspects, avoiding background
towards the edges, allow transparency and images in parts and last but not
least manage image sets).
The challenge here is a suitable GUI. This tool can then be used as a library
in aspects 1 and 3.
-- at 3: this is the most challenging, since the normal approaches with Java
are not useable (to catch mouse and keyboard actions you always need some
overlay frames, which are not possible in all cases (e.g. full screen) and all
over the place).
The most promising approach is using the JNativeHook library, that allows to
run a recorder in the background that tracks all mouse and keyboard actions and
decides wether it is a command for the recorder or going through to the running
application. In the end you have some raw workflow, that can be used as is or
optimised/augmented with the tool 1. The recorder also tracks screen content
using intermediate screenshots in various aspects depending on the state of
recorder and/or application. Images might be edited while recording or
afterwards based on the screenshots using tool 2.
Conclusion: I think your ideas go into the right direction and I am very
interested, to further discuss the requirements with you.
Maybe we can agree on some toolbox, that might be used in your Eclipse
based approach and in SikuliX2 as well.
You received this question notification because your team Sikuli Drivers
is an answer contact for Sikuli.
_______________________________________________
Mailing list: https://launchpad.net/~sikuli-driver
Post to : sikuli-driver@lists.launchpad.net
Unsubscribe : https://launchpad.net/~sikuli-driver
More help : https://help.launchpad.net/ListHelp
