In this article, discover our feedback on using two tools in order to implement end-to-end tests: Cypress vs Puppeteer. End-to-end testing is a methodology which is designed to ensure that an application is functioning properly, by automating browsers to run scenarios of typical actions made by end users.

• Puppeteer is a tool developed by Google that automates Chrome using their devtools-protocol.
• Cypress is a tool developed by a company of the same name. The test runner is open source and free. They also offer paid services that integrate with it.

The Test

To evaluate these tools we implemented a relatively complex end-to-end test using both tools. The test consisted of 89 steps. This was mostly clicking and typing, with some dragging and dropping. Neither solution ships with drag and drop functionality and we used roughly the same solution for both (creating events in JS and using dispatchEvent).

The Results

Speed – Tie

Cypress and Puppeteer both ran the test in roughly 35 seconds. So it seems neither has much of a speed advantage.

Reliability – Cypress

It became clear while working with both that Puppeteer was much less reliable. Sometimes clicks would not happen, and the tests would timeout. This happened in spite of the elements being present at the time the click was being attempted. It seemed to be a bug in Puppeteer but we did not have time to investigate it thoroughly.

Tooling – Cypress

Cypress was designed for E2E testing. To make Puppeteer run the same tests as Cypress we had to add a lot more functionality ourselves.

One of the fundamental differences between these two solutions is that Puppeteer is not a test runner, and is not even an end-to-end testing tool, it’s just a browser automation tool. To use it for testing we used jest-puppeteer, which worked relatively well. Cypress uses mocha internally as a test runner.

Aside from including a test runner Cypress also adds a shell around the application being tested. The shell includes:

• Logging of commands.
• Functionality for debugging tests.
• A tool to help find selectors for elements.
• Tools to inspect the DOM at any point during test runs.
• A view of the app that is automatically sized to the available area for debugging while still using a given resolution for testing.

Cypress also watches for changes to tests and will re-run them while you work. Given that Cypress is very thorough in cleaning up browser state between tests this is a huge workflow improvement. You can edit your test and almost instantly see it start running again with your changes.

API – Tie

The Cypress API provides more useful functions for E2E testing, including assertions, which once again are not included with Puppeteer.

The main thing we didn’t like about the Cypress API is that it uses a custom scheduler for its commands. cy.get(‘.button’).click() looks like a synchronous command, but it’s actually scheduling a command with Cypress. This is not intuitive and makes understanding the flow of Cypress tests slightly trickier than Puppeteer tests. There are architectural reasons for Cypress not using async/await but we’re hopeful those can be worked around eventually.

In defense of this approach we do find that it restricts how you can write your tests in a way that forces you to write more deterministic tests, which is usually better.

Conclusion

We found we liked using Cypress much more overall. It had better tools included, and it ran more reliably.