Test automation with a great new tool: Cypress
Early October 2017 saw the release of a new tool for test automation, Cypress 1.0. It's not just another tool, it's a tool with specific goals in mind. The main goal is to diminish the problems that most tools have with frontend testing. Problems like slowness of the tools and flakiness of the tests are common. This is causing a lot of maintenance of the tests and is therefore time-consuming, costly and demotivating.
Frontend testing
At bol.com we started out with QF-Test as the first general test automation tool. In itself not a bad tool, but with about 15 testers contributing to the test base, it soon spiraled out of control and many scenarios were written, basically doing the same thing. Then we thought it was a good idea to get developers involved as well. They opted for a tool based on Java, which they used for coding anyway. So we switched to FitNesse, which is based on Selenium. However, again the scenarios spiraled out of control, including duplicated scenarios. For frontend tests, FitNesse is quite slow in our experience.
Since we have to do frontend testing, being a webshop, we were keeping an eye out for better tools, and came across Cypress. It’s a relatively new tool that is recently released as a public beta version, even though development started 2,5 years before that.
Cypress deep dive
Cypress can be used for different types of tests, Unit, Integration and End to end tests. It is designed to work with JavaScript files, which is understood by both our frontend and backend developers. Therefore, they can all help writing tests. We have even seen some examples of Cypress being used for unit tests, as it's sometimes easier to write them in JavaScript then in Java, according to developers. This shows that Cypress is not simply a frontend testing tool, but that it can be widely used.
Ways of testing
There are two ways to use Cypress: a stand-alone and a command-line version.The stand-alone version can be used to write the tests and refine them. When you run the tests in the stand-alone mode, a browser is started and you can see what the commands of the tests are actually doing on the website. Once the test is finished, you can hover over a piece of the test that failed and see the status of the website at the time. That provides very quick feedback about what went wrong.
Cypress can also be run from the command-line. That is really handy if you want to run several test projects at once, without starting every project manually each time. When running from the command-line it automatically records videos of the test runs and presents the results of each test (when each test is done). It also gives an overview of the test results at the end of the test run. This means that you can have a visual check of the test runs at any time you want. Just open the video for a specific test you want to inspect.
Really, really fast
We compared running similar frontend tests in FitNesse and Cypress. Cypress was more than 5 times faster than FitNesse. It also automatically waits for all page elements to load, before checking if an element exists. Therefore, retries or waits in your scenarios are not necessary anymore. This can also speed up your tests considerably.
Another cool feature is that Cypress can run headless. This means that the browser itself is not started, but the HTML code of the site is used for the tests. That way tests can run faster.
Continuous Integration
We always wanted to incorporate a testing tool in the build, so you would know right away if your new or adapted code breaks your tests. FitNesse was too slow to be incorporated in the build. With Cypress we have a fast tool and we succeeded to integrate it in the build.
For now only Chrome and Electron (headless Chrome browser) are supported, but support for Firefox is already on the roadmap and in progress (Cypress roadmap).
Considering the fact that the current version (3.0.1) is still a beta version, it shows that Cypress is already useful and still getting better. We're eager to keep using the tool and see what will happen when it's no longer a beta tool, but a full-blown version!
If you want to try out this great new tool yourself, you can download it here: Cypress website.