Recently I have been appointed “Verification Leader, Core Foundation Team” whatever that means. Essentially it translates into taking care of functional testing. Ah, testing, well regarded, often ignored, always controversial.
“Testing” usually means “unit testing” to developers. Code is what we understand (I can’t speak Russian but I can speak C++ ðŸ™‚ ) “I’ve kept my API stable but I’ve optimized the internal implementation. What did I broke?” With proper unit testing, you can go for a coffee and when you are back answer to the question is on screen: “oh, after optimization I forgot about corner case X, I’ll fix it”.
So the code is right. As Knuth would say “Beware of bugs in the above code; I have only proved it correct, not tried it”.
Try. That’s the key. The user does not care whether the application is implemented in C++2011 using lambdas, concepts, variadic templates and throws exceptions across shared libraries, or in plain old Visual Basic 6. Does it work? Does it load the file when I click File -> Open? Will it show a computer when I click and drag the computer icon from the palette to the canvas? That’s what matters.
In KDE we have unit tests in kdelibs (in fact it’s a requirement to get code in) and there are a few UI tests for the KDE User Interface module, which contains essentially widgets for use with your KDE-based application.
What about applications? What about Amarok, Gwenview, Kexi, Okular, etc? How do we know they work fine? How do we know they actions do what they should do and the application renders as it should? Correct me if I’m wrong but I think currently we rely on the input from our users: when they report something’s misbehaving, we oblige.
There are many tools for automated GUI testing. Some are open source, like Xnee or White, others are closed-source, like Froglogic Squish and SmartBear TestComplete, which also happen to have the best support for Qt-based applications.
Are those tools useful? Why, yes, they are. For instance, there is a well-known bug in KDE Windows which makes Digikam on Windows show empty context menus when the Qt style is set to CDE (the default; if you use the Oxygen style everything looks fine). This kind of bugs can be easily detected by simulating user actions, then OCR‘ing the text of the popup menu. Or even just telling the automatic UI test to take a screenshot of the popup, then comparing pictures. Same for PDFs that render oddly in Okular on some Linux distribution due to fonts, FreeType configuration, an old version of Poppler, etc
The tool I am most familiar with is TestComplete, which suffers a bit with alien widgets (you need to build Qt without accessibility support). I have been told Squish is much better but given that there is no freely downloadable version and it does not support WPF (a requirement for the project I’m working on now), I can’t try it.
Here is a wish for KDE users who want to join development but have little or no programming skills: go to SmartBear’s website, download TestComplete and record/write a few UI tests. Let us know (in the forums, in the mailing lists) how far you got. Maybe we can make UI testing -a must in the corporate software development- part of the KDE development and release process.