How to Structure Visual Regression Testing Based On Business Needs

In my previous article, I discussed the basic concepts of Visual Regression Testing. Now, let’s delve deeper into defining a visual testing architecture based on a business model.

Since technology is constantly changing, architectural decisions require a lot of careful consideration. We need to think about how sustainable, maintainable, and scalable the source structure is in order to make a tailored decision. In this article, we’re skipping the technology decision and focusing on the visual testing architecture decision. We’ll take a look at two case studies, go through the design thinking process, and end up with a solution for each case.

Identifying the Business Side of Things

The first step is to analyze the context of the project and see which visual testing architecture fits best. Start by interviewing your PO, or whoever best knows the visual regression testing needs of the project.

Here are a few questions that should help you gather information. These questions can serve as a base, but make sure to also include any questions that are specific to your project.

1. How often do we release?
2. How often do we change the CSS?
3. How are the main browsers and resolutions?
4. Is the website responsive?
5. Is the website international? If so, for how many languages?
6. What are the most important pages/features/flows that directly affect revenue or impact the final user?
7. List the 3 major features released last time.
8. List some of the bugs we caught in production, preferably related to visual changes.

With those questions answered, we’re ready to move forward!

Brainstorming the Architecture

Now that we’re familiar with the business, we can start thinking about the visual testing architecture. For this, we also need to have some questions answered:

1. What technologies do the project use?
2. What technologies should I use?
3. What tests are already implemented?
4. Should I use the same functional tests and just add page checks?
5. Where should I add them in the pipeline?
6. Considering the tests pyramid, which tests should I add at this level?

These are some of the questions you can ask yourself to help figure out the best visual testing architecture for your business needs.

Proposing A Solution

The next step would be to analyze the information acquired and come up with a solution.

At this point, you’ll be able to define:

1. Whether or not it’s worth implementing visual regression testing for this project
2. Which features should go through visual regression testing
3. Which browsers you should cover
4. Whether or not you should re-use the functional tests structure (since the coding is pretty similar)

Now, let’s get to some real cases and put the steps together.

Hands On

Here are two case studies we’re going to use as examples. Take some time to go through them in order to gain a better understanding of the information gathered from steps 1 and 2, as well as the proposed solutions in step 3.

Case Study 1

Here’s the URL for the first case study: https://www.avenuecode.com/

Information gathered:

• There’s only one website under the project structure.
• There are very few external integrations (external services or database).
• The project has a simple and stable structure.
• The releases are usually done once a month with few CSS changes.
• The website has internationalization in 2 different languages.
• The website is responsive, and the top browsers are Chrome and Safari.
• The technologies the project use are HTML and Javascript (no frameworks), with no functional automated testing.
• The focus is to guarantee that the changes don’t break cross-browser compatibility.

By analyzing the business, we can conclude that there’s no big need to conduct visual regression tests because there aren’t a lot of CSS changes. However, if the company allows for investment, we can run a check for the most important components such as headers, menus, footers, contact page, and careers page for a desktop and mobile browser. The visual regression test source code will be different from the functional tests. Check out the second case study for more details.

Case Study 2

Here are the URLs for the second case study: https://www.fansedge.com/ and https://www.collegebasketballstore.com/

Information gathered:

• Different brands are using the same platform, so there are different websites with similar structures under the same project structure.
• There are a large number of integrations such as internal and external services integration and CMS (content management system).
• The project has a robust, dynamic structure.
• The releases are at least twice a week with a lot of CSS changes.
• The websites don’t have internationalization.
• The websites are responsive and their top browsers are: Mobile Safari, Chrome, and Chrome Mobile.
• The goal is to guarantee that changes to one site don’t impact other sites, as well as guaranteeing that the changes don’t break cross-browser compatibility.
• The websites have 90+% code coverage on unit tests, and 80+% of the P0 functional automated written using Webdriver.io.

By analyzing the business side, we’re able to see a high need for visual regression tests because there are large amounts of CSS changes, some of them specific to only one or two brands. One of the challenges is working with dynamic areas, so we need to have a large coverage with regards to visual regression testing — mainly component oriented, since we reuse components throughout many pages. We also need to be able to work with different browsers and resolutions, such as for tablets and mobile devices. The number of different websites is projected to increase approximately every quarter of the year as well.

For both cases, we decided on decoupling the visual regression testing from the functional testing for the following reasons:

1. Different technology
   Visual regression testing and functional testing each have their own technologies that best fit each context.
2. Different teams
   There might be different teams working on each layer of tests; therefore it can become easier to manage permissions, merges, tasks, etc.
3. Different pipeline stages
   We might want to use tests in different stages of the pipeline, such as running them only on Friday nights, or whenever we have releases.
4. Different target browsers
   Depending on the project structure, we might want use different target browsers for visual regression and functional tests.

Conclusion

To sum up, we went through the steps proposed to build a good architecture solution by identifying the business needs and brainstorming the architecture. Now that we understand the needs, it’s time to build some examples and evaluate the results for your particular situation.

In the next article, we’ll implement the topics we just discussed in a visual regression test project for these two use cases. I’d love to hear back from you! Let me know if you found this article to be helpful, and feel free to send over any questions you might have.

Originally published at https://blog.avenuecode.com on November 7, 2017.