This page was machine-translated from Portuguese and may contain mistakes. Read the original
In this Tech Chat, I welcomed Jess. She’s also part of the Playwright WhatsApp group and tells us a little about what the migration process from Katalon (Selenium) to Playwright was like—and who better than her to introduce herself to us:
"I’m Jessica Silva, 30 years old, and I’ve been working in quality assurance since 2015. I have a degree in Accounting, and my first job was at a tech company working as a BA (or business analyst) on a public accounting system. Until then, I didn’t know anything about development or tech terms. The funny thing about all this is that I always felt I wasn’t studying what I wanted to, since I started college at 17 without much guidance on what to major in. I spent four long years studying accounting before eventually finding my way into the tech field. I think I had to go through this whole journey just to finally understand my purpose.
I switched fields to IT, and during that transition, I studied Software Engineering at PUC Minas to further develop my skills in the field. I worked for about seven years at the company where I began as a QA engineer until I decided to do an exchange program in England in 2020, and a year later, in 2021, I decided to move to Portugal to dive into the tech market. I’ve worked on various projects in Europe since then; however, as fate would have it, I’ve also been working on projects in Africa.
Today, I am the QA Tech Leader of the automation team and have been helping to train Angolan professionals in the field of test automation. It’s been a unique experience to bring my knowledge to a country on the rise in the tech sector, and my goal is to be a pioneer in implementing and training professionals to use the Playwright tool and the QAOps methodology in Angola."
What an inspiration, Jess! I’m so grateful to have met you through the group. Thank you for participating in this Tech Chat!
1. What was the old framework, and what was the main problem you had with it?
The tool we used to implement automated tests was Katalon, a paid tool based on Selenium. We used the tool’s scripting feature, which only supports the Groovy/Java languages. Our biggest and main problem was the tool’s limitation in terms of the freedom to choose the programming language, structure folders, and use the Page Objects pattern, as well as in debugging capabilities, since the tool isn’t open source and restricts access to essential features. The initial idea behind using Katalon was that most QA testers were manual testers with no automation experience; thus, no technical knowledge of Katalon was required, and they initially used the tool to record the screen and generate scripts.
2. How did the idea to migrate to another framework come about?
Given the tool’s various limitations and its slowness—due to a heavy interface based on the Eclipse IDE—we could see a gap in the team’s delivery speed, as well as more issues related to the tool itself than to the code or the system in question, not to mention that the tests took about 5 to 10 minutes to run completely.
Consequently, at that time I took on the role of Test Architect and immediately suggested to leadership that we migrate to an open-source tool to give us more autonomy and agility in test automation.
3. How did the decision-making process for Playwright go?
On the last project I worked on in Portugal, the tech stack was based on Playwright, and I had the opportunity to participate in the tool selection process as well as in the migration from the previous framework, Taiko Gauge, to Playwright. I could already see in the first month of migration that the developers faced a very steep learning curve with the tool, as well as challenges in the speed of migrating tests written in another language to TypeScript in Playwright. So, for this new project here in Africa, I suggested using Playwright and conducted a proof of concept (POC) comparing the main tools—Robot, Cypress, and Playwright—to determine which one would best suit the project based on the system’s architecture. Consequently, after creating the proof of concept, I was invited to take over leadership of the automation team and begin a restructuring process in the department. The team had no experience with any of the tools used in the POC, so a mentoring and training process was initiated to eventually implement Playwright, since I was the only one who had a good command of it and had prior experience working on a migration to Playwright.
4. Playwright supports several languages. Which language do you use, and what led you to choose it?
We use Playwright with TypeScript. We chose it because of the language’s gentle learning curve, and because the team already had knowledge of JavaScript and experience using Node.js, which greatly simplified the tool’s configuration and use of its features.
5. How did you structure the migration (in terms of the timeframe, who would participate in the project, and which test prioritization technique was used)?
After restructuring our processes, we designed a migration plan that called for a complete migration to be completed within three months, including hiring, onboarding, training, and the implementation of CI/CD. Thus, in terms of testing, we prioritized functional and/or unit tests first so that, as we migrated our methods, we could create unit tests to validate the most essential parts of our code. We divided up the tasks and invited a developer and a DevOps engineer to participate in the migration and assist with the continuous integration processes, which was extremely valuable. In total, five people participated in the tool’s restructuring and migration, with three directly involved in the code migration and two in CI/CD.
One concern we had to address was the different projects and their unification in Playwright. In Katalon, we had to create a project for each system and also split them in the remote repository. In Playwright, however, we were able to configure the separation of systems in the playwright.config.ts file quickly and without complications, and we used all systems within a single remote repository.
6. When you began the migration, how long did it take to see significant results?
The migration reached a significant milestone as early as the first month, when the first suite of critical regression tests was migrated in just one week. The response was so fast that by the end of the first month, we had already begun addressing a backlog of new, unforeseen requirements and completed the migration. Thus, the migration—which was originally projected to take 3 months—lasted about 1 month, including onboarding, training, and the migration itself.
7. What was the most valuable outcome of adopting Playwright?
The biggest benefit was the increased agility and closing the gap we had in the team’s delivery speed, in addition to the satisfaction reported by QA and Dev team members during 1:1 meetings and daily stand-ups. There was also a dramatic improvement in the total execution time of end-to-end (e2e) tests, which used to take 5–10 minutes and were reduced to approximately 1 minute each.
8. What feature do you like most about Playwright?
Without a doubt, the various debugging options, as well as the trace feature and the flexibility to configure the tool according to our specific projects.
9. What’s still not quite right with the testing project after the migration?
I believe we have a lot of room for improvement, since after the migration we’ve been performing several code refactorings, which is normal at this stage.
However, our API tests are a point of concern, since the systems use the SOAP protocol to communicate, and the automation of these test cases is done in the SoapUI tool using Groovy. I believe that in the near future we’ll need to come up with a solution to use the collection of these tests integrated into Playwright without losing this automation. In terms of CI/CD, we were able to integrate with the Docker image; however, for local execution, we haven’t found a way to integrate with Playwright as a third-party tool to generate some precondition data before proceeding with frontend testing.
10. Is there anything you would do differently?
Not at the moment. I believe that, based on the restructuring and the team’s openness to making it work, we did exactly what needed to be done at the right time and with the appropriate techniques. However, I believe this question will arise once we’ve had about a year of migration and use of the tool.
11. Did you use (or do you plan to use) artificial intelligence at any stage of the migration? What is your opinion on this?
Yes, we even used AI as a learning tool to interpret our code developed in Groovy and convert it to TypeScript, in order to reinforce our understanding and highlight the main differences and benefits. It was a good experience for the team, since many found it easy to read code written in Groovy but initially had questions about TypeScript’s typing system.
My honest opinion on this point is that if we learn to give the AI the necessary commands and keep in mind that it’s a tool to assist us in learning, it’s well worth using. However, if we reverse the situation and use it as a solution to our problems, we may miss the main point of gaining knowledge—and end up generating incorrect code and unwanted bugs in the project.
12. What’s the best piece of advice you’d give to someone thinking about migrating from another framework to Playwright?
First, determine whether it’s the right tool for your project’s purpose; analyze the architecture and key aspects of the system to verify that Playwright would be the ideal tool for test automation based on the system’s interface. Playwright is a very powerful tool; however, for some legacy tools or those with older component interfaces—such as PrimeFaces—jsf, Playwright wouldn’t perform very well in terms of page performance—not because of the tool itself, but due to the components and how pages are loaded, which would significantly hinder test automation. Of course, with a solid proof of concept, it would be easy to identify these points of concern.
This situation happened to me, and the proof of concept was taken into account by the development team when they decided to migrate the web application to Angular—which will be extremely powerful for using Playwright, since it uses TypeScript in the frontend component. This might be a golden tip for thinking outside the box and helping to improve the project’s frontend as well.
Additional References:
- Other Tech Chats
- Stefan Teixeira’s Talk on Playwright at the Minas Testing Conference 2022
- Playwright Training Sessions by Renata Andrade
