This page was machine-translated from Portuguese and may contain mistakes. Read the original
It’s very common to get questions like “Is Playwright better than Cypress?”, “Should I switch from Selenium to Playwright?”, “Does Playwright replace Robot?”.
The answer to all these questions is: It depends.
No software is better than another in isolation.
If you have a suite of 1,000 tests running with Cypress and the tests are stable, the pipeline runs in a timeframe that’s acceptable for the project, and the team is productive, you MIGHT not need to switch to a “more modern” framework.
In technology, there will always be something “better” available at any given moment. But better for whom? Better compared to what? It’s always good to keep up with updates to frameworks, practices, and methodologies, but at the same time, make smart decisions to build solutions that are sustainable for the teams.
There’s a (free) course by Angie Jones on how to lay a solid foundation for automation, and one of the topics is how to choose a framework (it’s important to note that this isn’t the only factor, okay?!). Building on that course, I’d like to share with you here a POC template that worked very well for me.
I’d also like to remind you here that AI should be used with GREAT moderation and wisdom, as it can yield incomplete or outdated results. Always consult the most up-to-date official documentation for the frameworks.
First, I’ll share my scenario so you can get a sense of my POC process:
I was a Lead Quality Engineer and was hired to build a quality team at the company; there were no other QEs. There were several projects, and the one I was assigned as a priority had a React front end with JavaScript migrating to TypeScript and had no end-to-end (e2e) tests. The back end was in PHP, and the developers were full-stack. They were all comfortable developing in JavaScript and/or TypeScript.
Personally, I believe it’s important to choose a framework that supports a language the developers are familiar with, as they can help with technical programming questions and code reviews. Additionally, I like to adopt a work model where developers have the freedom to implement or maintain the tests. This way, if the company can’t hire an optimal number of QA engineers (or for any other reason), we can blend the team and ensure the success of automation by having the QA engineers act as “quality mentors” rather than “quality guarantors.” There are other reasons as well, but let’s stay on topic for this article.
POC Steps:
1. Identify the reason for choosing the framework
If the team has no automation in place, I consider this the easier scenario, since you only need to list what features the framework must support.
If the team already has automation in place, it’s important to identify the reason for changing the framework. Is the automation flaky? Is it complex to maintain? Is it slow to run? Is there a lack of a testing environment?
This step is important because it will guide the list for the next step. Sometimes the project just needs refactoring to solve problems like flakiness or maintainability, or to integrate a newer library to resolve a specific issue with reporting or parallelization in the pipeline.
I recommend involving professionals from other areas, such as DevOps and architects within the company, to help find an effective solution to the problem. If this isn’t possible, technology events are excellent sources of knowledge; many professionals share how they’ve solved problems, and someone may have done something similar and be able to guide you better.
In either case, if the decision is to adopt a new framework, list what the framework needs to have to solve the project’s problem—for example (these are just examples; it doesn’t necessarily need to have all of them):
- support for a specific language or multiple languages (in case future migration is needed)—Playwright, for example, supports Node.js, Python, Java, and .NET, though the languages do not offer the same features
- support for cross-browser testing
- a good variety of selectors presented in a user-friendly way
- flexible architecture (you can use POM or not; you can create customizable commands)
- Native parallelism
- API integration (to mock third-party services or intercept requests)
- Supports cloud testing (BrowserStack, LambdaTest, SauceLabs, etc.)
- Supports visual regression testing
- Provides test reports
- Accessibility support
- Performance support (web and API)
2. List the frameworks evaluated
In this step, we’ll identify frameworks that solve as many of the problems we listed in step 1 as possible.
I recommend reaching out to your team and neighboring teams to gather suggestions for existing frameworks, rather than just sticking with what you already know. Also research the most widely used frameworks, the latest releases, etc.
If necessary, hold a simple vote on the suggested frameworks and select 4—or at most 5—for an initial evaluation; at this stage, you only need to read the main documentation and understand how the framework behaves. Consider the following points:
- the project’s GitHub page and see how many stars it has
- the number of releases and their frequency
- the number of bugs
- npmtrends
- Community support
After understanding what each framework offers, narrow down the list to 2, or at most 3, to implement the POC.
3. Choose what to implement
Given the nature of your project, choose a feature that is simple (so you don’t spend too much time on it) and, at the same time, encompasses at least one or two of your project’s key characteristics. Involve your Product Manager to determine which feature is most frequently used in the system—for example:
- Login and Product Registration
- Product Search and Purchase
Ultimately, this will vary quite a bit, but the idea here is to test how easy it is to use the framework’s features for the tasks you’ll be performing most frequently in the application. Often, a simple login feature alone won’t cover everything. At the same time, don’t choose a feature that’s too complex, since the POC needs to be completed relatively quickly. I’d say 1 to 2 weeks would be ideal, depending on the project and the time you have to dedicate to it (sometimes you’ll only have 1 or 2 days ¯_(ツ)_/¯).
4. Define the evaluation criteria
In this step, it’s important to tie in what was listed in Step 1, “reason for choosing the framework.” If you’re looking for a framework that runs faster, performance should be a criterion. However, remember that some criteria are interrelated; for example, performance can also be achieved through parallelization. So defining the criteria shouldn’t be a mechanical process. There are objective criteria as well as subjective (combined) criteria. Approach this step with great care.
Some criteria I usually evaluate are:
- language support (depending on the current team’s expertise and the ease of finding professionals in the market)
- performance (for implementation, execution, debugging, updating, and report generation)
- parallelism (not only per file, but per test within the file)
- variety of selectors (especially support for user-facing selectors and auto-waiting)
- flexible architecture
- support for API interactions (third-party mocks and request interception)
- mobile support (or integration with other modern frameworks)
- support for cloud testing
- ease of installation and use
- objectivity of documentation (how much time do you spend finding information)
- Support for identifying issues (does it provide reports? Are errors clearly displayed in the console? Does it provide a trace? Does it provide screenshots? Does it provide video?)
You should decide together with your team which criteria to adopt and how each one will be evaluated.
5. Implement the code in the chosen frameworks
Install the chosen frameworks from scratch and implement the defined scenarios.
It’s extremely important at this stage to start evaluating this process, even if it isn’t a defined criterion. Your team’s day-to-day work will involve implementing new tests, so it’s important that the implementation process be simple.
Pay close attention to everything you do in this step:
- how many external packages/libraries you need to install
- how long the installation takes (for the framework itself and the additional libraries)
- how long it takes to find guidance in the documentation
- how much you can use global settings in the configuration file
- how much you need to write in the test files (and repeat information across different files)
- how easy it is to write and run the test
- How easy it is to understand errors during the implementation process
6. Fill in the evaluation criteria
I usually start this step as soon as the criteria are defined. I fill them out as I go along; I never leave everything for the end. Here, I use an Excel/Sheets spreadsheet with scores and comments for each criterion by framework. Simple and to the point. The team should trust the person leading the process without needing to justify every step in excessive detail.
It’s normal to add and remove criteria during the process; it’s normal to assign a score and then change it later. It really is a cyclical and interactive process.
If you’re conducting the POC with others, it’s very important to align on the evaluation method—discuss how you’ll assign scores and try to find common ground to calibrate the ratings. Personally, I prefer to conduct framework POCs individually to minimize noise.
It’s also a good idea to discuss part of the process with a few people to see if there are any points you may have overlooked. Choose someone you trust, and be prepared to set aside suggestions and opinions that you deem invalid for your specific context.
7. Create a DMF and submit it to the team for review
A DMF (Decision Making Framework) is a methodology that provides a detailed breakdown of each evaluation criterion, facilitating collaborative decision-making and clear communication between decision-makers and their teams.
If you choose to use this format, let everyone involved (including leaders and affected teams) know that this process will be adopted. Another less disruptive and more traditional alternative is to put together a presentation with the results and invite those involved. I prefer the DMF model because it fosters greater involvement and participation, encouraging team members to exercise autonomy and stay engaged.
I use Notion or Excel itself. You can also do this in Confluence. Depending on the tools you use at work, create a document with the information below (and anything else you find important).
There are several formats; I’ve been using something like this:
- Overview: what’s being done and what the goal of this DMF is. Keep it concise. The reader should be able to tell whether this DMF is relevant to them or not.
- Context: a detailed overview of the activity. The reason for this decision, its impact, and links to prior discussions or data that led to this need.
- Evaluation process: explain how the evaluation is being conducted (whether by one or more people), the criteria to be evaluated, and links to the document containing the scores and to the source code repositories.
- Pros and Cons of Each Framework: At this stage, I aim to highlight the framework that received the highest score to justify the decision. The work has already been done; the purpose of this document is to help the team understand what was accomplished and to identify any risks that may not have been considered. Include screenshots and any figures that stood out significantly.
- Proposal: In one sentence, state the decision that needs to be made. Ex: “Adopt the Playwright framework as the UI and API automation tool for the Death Start project.”
- Voting: Add the people who should be involved in this decision. Leaders, New Hires, Interns, DevOps, Developers, Quality Engineers, Support Team. Include between 3 and 7 people who make sense for this decision.
- Voting deadline: Usually, one week after the DMF is created is sufficient. You can schedule conversations to discuss any unclear points so that a person can make an informed decision. It’s also possible to have a discussion with anyone who votes against the proposal until they feel comfortable agreeing with it. Or, in the event of a tie, you can designate someone to make the final decision.
- Decision: What was decided (to be filled in once the voting period ends).
The goal of this step is to officially decide on the framework to be adopted.
Example of a DMF I created to evaluate Playwright vs. Cypress.
Conclusion
By following these steps, it is possible to make decisions that are appropriate for the entire development team, providing visibility into the quality team’s work and bringing together the areas impacted by this process.
Remember to establish an ongoing communication strategy regarding the status of the new framework’s implementation and the results achieved. In addition, keep track of new framework releases and take advantage of improvements, keeping the version up to date whenever possible.
Additional References
Playwright: https://playwright.dev/
Cypress vs. Playwright Code Battle at the Minas Testing Conference 2023: https://testingwithrenata.com/portfolio/minas-testing-conference-2023/
Answering Questions from the Code Battle:
Part 1: https://testingwithrenata.com/portfolio/youtube-live-mtc-2023-parte-1/
Part 2: https://testingwithrenata.com/portfolio/youtube-live-mtc-2023-parte-2/
