Hi Ian, I wrote way way too much, please feel free to edit/reword/chop to your hearts content. If the example isn’t strong enough I’m happy to pick another..
PS. turns out about 9 months ago I added a hybrid example using Watir-WebDriver and Sikuli onto github-
Some contexts where I’ve used Sikuli and found it to be helpful:
- Native windows invoked from a web browser eg. Print dialogs, or save as.
- PDF File contents opened in a 3rd party application
- Image content testing eg. search for a particular image on a search engine, ensure you actually got the right image.
- Mixed technology applications where no one tool can drive through the whole scenario.
- Win32 (C++/Delphi)/.NET UI with lots of custom controls
- Citrix hosted apps (its all one great big image anyway)
One particular horror story was a Delphi app that was in a bit of a mess- it was implementing win32 controls, as well as custom ones that had been bought from a vendor, both had been abstracted many times, with the parent control behaviours being concealed through the abstraction process. Interrogating the UI via COM returned next to nothing, and using .NET to walk the UI would cause .NET to throw and exception and give up! This eliminated using .NET’s UI.Automation and therefore Project White as options.
In this case, I quickly was finding that using native tools wasn’t working and I was under pressure to turn something around.
My first step, was to communicate to the team that building automation for an app that was clearly never intended to be automated wasnt working. They needed to take responsibility for it and start refactoring the code, fix the lack of testability and add unit tests as they went as the test automation wouldnt be sustainable even with Sikuli as a work around. Meanwhile I’d build a small test suite using Sikuli just to get them past it and to give them some coverage while they re-factored.
Shortly after this with some code changes were made that unblocked COM access after which I ended up pairing up with another dev and implementing automation using COM as our entry point.
It wasnt smooth sailing even with this approach, but years later it was still being used even though that option was intended just as a stop-gap measure until the app was re-written (which back then was going to be a near-term event).
The entire time we depended on Sikuli was around 2 weeks.
The lessons learned in that time were that Sikuli:
* Very good at matching images
* Quick once its running, in some cases faster than native calls without caching.
* Can drive UI that otherwise would take you weeks to figure out how to call.
* It is reliable once the app you’re testing has stablised.
* Seldom crashes
* Can be used to build scenarios from screenshots until the actual software is delivered allowing some degree of ATDD to be undertaken then adjusted accordingly.
* It can type, and click, but cant read from the target application. It compares one image, with the current one. eg. dynamic values = major problem. Consider OCR or other potential work arounds- but none of it is attractive.
* Other applications and dialogs that sit on top of, and therefore obscure view of your app will break the automation. Adding code that finds the window and brings it to front on every interaction is one work around, but its annoying to have to do it.
* Compiling the required libraries for OpenCV and other dependencies can be painful to complete especially as they get older.
* Complex applications eg. scrolling up/down or across quickly blow out the complexity of the scripts.
* Small UI changes can and often do cause mass re-recording of images captured. If not done carefully can completely invalidate the test scenario.
* Difficult to debug and time consuming as its a “change” then “re-run” and “check” model.
* Cant be run headlessly unless you’re running a session in a framebuffer (Xvfb is your friend) or other similarly hidden away interactive session. Creates challenges when executing tests on CI boxes that have no window manager.
* Difficult to parallel execute tests, due to Sikuli needing to point and click with absolute control.