Test Automation Foundations

In the previous chapter, we have learned the key objectives of test automation as well as the potential benefits and risks that come with it. In order to succeed in using test automation, it is important to consider all the factors that influence your implementation. This chapter will walk you through the critical factors that might influence your success.

There are many factors that influence test automation. In this chapter, we categorized them as organizational factors and application under the test's environmental factors. First, let's go through organizational factors. Organizational factors refer to those within the company or the team itself, namely the background skills and knowledge of your testers.

For instance, if you're a manual tester, using an automation library, like Selenium, to kickstart a test automation framework will be problematic. In this case, intensive technical knowledge and advanced programming skills are required. Keep in mind that the purpose of implementing test automation is to test other software products.

So, if you use a test automation tool made by a third party like Katalon studio, manual work will be reduced significantly. All you really need to do is manage and generate test cases. As you become more experienced, you can start building your own test cases or test scripts, as tools like this do support script extension to build custom keywords.

Now let's go into details about the application under test and its environmental factors. When evaluating the context of the application under test and its environment, factors that influence test automation may consist of the interfaces, architecture, sizes, and complexity of the application under test, as well as third-party software.

First, for the application under test interface, the user interface isn't the only one out there that needs to be considered. Think about API, CLI, or voice command as well. Since no test automation tool can work on all interfaces, as a tester, you should carefully consider the right one to pick. Second, your system's architecture also impacts your test automation solution. For instance, a monolithic architecture surely needs a different approach compared to a Microservice architecture. That's why you need a different approach for each type of architecture.

Next up, how about we mention the size and complexity of the application under test? If your system is small and simple, a framework that is meant for a more complicated approach wouldn't work and vice versa. For instance, when you need to test various steps and features of your complex web application such as online payment, a simple test automation tool might not be able to help. You might attempt to integrate many tools to try on your application such as test design tools, test execution tools, and test reports, all from multiple vendors.

But what if these tools do not ensure interoperability? In order to solve this problem, you should choose a QA Orchestration (eco-system) tool that includes multiple solutions like Katalon, they have Katalon Studio to design tests easily and quickly, Katalon Test Engine to run tests from CI tools, and TestOps for reporting.

Lastly, we often don't only build our products but get integrated with third-party software as well. Oftentimes, complex systems consist of various components including third-party software, both internal and external. Because they are managed by third parties, they might be changed suddenly. That might affect your systems or at least, a feature in your systems, so you should consider test automation that helps prevent or detect these problems earlier. Thus, automated integration testing or contract testing should be supported by your test automation solution.

Choosing the wrong test automation solution might make the job even more difficult, which defeats the purpose of using test automation in the first place. Placed in different contexts, you need to consider all key characteristics of the application under test, also the maturity of your team to pick the most suitable solution.

Welcome back to Katalon course. In this chapter, we will dive into basic concepts of the Test Automation Pyramid, and the strategy to apply various types of tests and testing frequency in each level. Before launching to the market, every software system needs testing at different levels to ensure its high quality for specific objectives. Typical test levels consist of, from the pyramid base to the top: unit testing, integration testing - or services testing, system testing, and acceptance testing. All will be discussed later in this chapter.

Generally, unit testing comes in the first stage where modules, features, and functions are separately tested. Integration testing takes place in the next stage to test the interaction between those functions or systems, which is why they are often called system integration testing. Next up, system testing is where the complete system is validated for both functional and non-functional aspects before moving to the last level. User acceptance testing conducted by clients or end-users to validate whether the application meets their expectation. Although all of these testing activities can be done manually or in automation, acceptance testing is preferred to conduct manually.

Test automation pyramid was first introduced by Mike Cohn in his book ''Succeeding with Agile.'' It emphasizes having a large number of tests at the lower levels (bottom of the pyramid) and the number of tests decreases as development moves to the upper levels. In Agile projects, this concept is also referred to as shift-left, where defects are eliminated as early as possible in the life cycle.

Before getting more in-depth about unit testing, let's determine the definition of a unit. In short, a unit is a piece of functionality you can test independently on multiple aspects and functions. For instance, each unit corresponds to each small element of the LEGO duck which can be separated. To be more specific, unit tests call a function with different parameters and receive expected values. In this process, external dependencies are removed by applying a test implementation or a mock object created by a testing framework. Thus what you need here is a suitable unit testing framework supporting your programming languages.

Why do we need unit testing in agile projects? Yet, introducing regression in the Agile method is at high risk due to extensive code churn. To prevent software build from breaking, development teams usually run automated unit tests before the source code is checked into the mainline (e.g. UAT or Main) of the CM system (e.g. Github or Bitbucket). Keep in mind that the automated unit test results provide immediate feedback on code and build quality, but not on product quality.

Moving up to the next level, we will talk about integration testing, a type of software testing focusing on components or systems interactions. This level is often called ''service testing'', which implies that the testing action is performed in the interfaces to test the interactions of integrated services. There are many ways to define integration testing, hence asking yourself what exactly you need to test is important. Perhaps, your demand is to call other application's APIs, the communication between modules within a system, or even operations between hardware and software.

There are two levels: component integration testing and system integration testing. Put it simply, let's say your system comprises two modules, A and B. After finishing unit tests, you will start verifying if these two modules communicate properly in both functional and non-functional aspects. It's a different story when you have to test between two different systems. For example, after your tests are executed in Katalon Studio, the testing reports are automatically sent to you through TestOps, which is an external service. Testing the communication between these two systems is called system integration testing.

As mentioned above, each test level has its own objective. To begin with, integration testing helps ensure the effective interaction between internal and external services, whereas unit testing verifies individual service or function. However, integration testing requires high effort since you have to merge related elements and deploy them to a test environment. This is true for both manual and automated.

There's one solution to handle the integration testing problem, that is contract testing. It is a methodology for ensuring that two separate systems (such as two microservices) are compatible with one other. It also captures the interactions exchanged between each service, stores them in a contract, then verifies that both parties adhere together.

Coming to the next level, it's now time for user interface testing, often named UI testing. With web interfaces, your UI testing can achieve two objectives. The first one is to make sure that your application's presentation layer works well across: multiple browsers (such as Chrome, Microsoft Edge), various devices (such as PC, tablet, and mobile devices that have different screen sizes), or different platforms (such as iOS, and Android). Now, you may think about visual testing and yes, there are tools to support this testing type.

The second objective that we need to mention here is the UI end-to-end testing, which mainly aims to verify the business processes and workflows of the application under test. For the fact that you need to deploy the complete application under test to the test environment to testing, this is definitely a costly process. Moreover, you might need plenty of end-to-end test scenarios, from simple to complex, to satisfy the functionality and non-functionality of the application under test.

Besides manual testing, regression test suites are usually automated to run faster, in parallel, and consistently. And among popular frameworks, Selenium has constantly been among the top choices when it comes to UI end-to-end testing. With Selenium, you can pick up any browser and let it automate your website, submit various data, and update changes in the user interface. This kind of automated test allows tests to be run on a Driver, or in headless mode. A headless browser is a tool for automated testing and server environments in which without a visible UI shell you can integrate your tests into CI/CD tools. Nowadays, almost all browsers can support headless mode.

When working with UI automated testing, you may face many obstacles. Typically, the 3 common challenges are:
- Applications change frequently - for instance, your competitors have just released a new feature or there are new technologies to adopt into your current application.
- Different ways in error handling in the same system - For example, in the same web system, there are some screens/pages that use inline error messages while others use dialog or js popup to show error messages.
- And, cross-browser testing - with websites, you need to make sure that your system works well on different supported browsers such as Chrome, Firefox, and IE11.

To resolve these obstacles, you need to have a solid test automation solution that requires your team's intensive automation testing knowledge and programming skills. Luckily, there are codeless UI testing solutions available for you, such as Katalon Studio. With it, you are able to test your web, desktop, and mobile applications without a single line of code.

So, in a nutshell, we've walked you through the overall concept of the test automation pyramid, also its benefits and considerations for applying test automation. Though detecting new bugs in your system is not only the main responsibility, it does verify that your existing features are functioning well, especially when code changes are too frequent. Remember that regression testing should be automated to gain a higher ROI.

Glad to see you again! It's now time to understand the test automation structure to effectively organize test cases. This is very helpful if you wanna build your own test automation tool.

A test automation engineer has the role from designing, developing, implementing to maintaining test automation solutions. In this chapter, you'll explore the generic test automation architecture along with its four layers, including test generation, test definition, test execution, and test adaptation. Let's examine the first layer of the architecture which is test generation, where you design, create, or generate automated test cases.

You can script test cases using any programming languages such as Java and C-Sharp in an integrated development environment such as VS Code or IntelliJ, but this relevant coding knowledge is needed. That said, how about manual testers who lack coding skills? In this case, your test generation layer should have a user interface to support test automation engineers. Thus you can easily drag and drop or select test steps, actions, etc. to design a test case with some clicks.

On the other hand, the test generation layer should automatically generate automated tests by transferring manual steps to a script. In addition, with tools like Katalon Studio, the test generation layer can even record manual steps on the application under test to automatically generate test scripts. So, after preparing your automated test cases, how will you manage them, and test data efficiently? Suppose that you want to execute a test case with a wide range of test data. Should your test cases be put in any specific test suite, such as confirmation or regression test suites?

Moving on to the test definition layer, it is where the test automation solution supports the definition and implementation of test cases and test suites. Usually, it separates the test definition from the application under test in terms of source code.

In fact, teams often ignore the importance of the test definition layer once they don't have as many test cases as they expect. In another aspect, the layer contains some means to take over both high and low-level tests, assigning different test data to the same test case, also known as data-driven testing. Besides, it handles the test procedures and test library components to verify a large volume of data combinations of each test case.

Moving on to the next layer is the test execution layer which supports executing test cases and logging automatically or semi-automatically. In this context, what do we mean by ''test logging?'' The terms mean that, it records and updates the pass or failed status of history logging. The layer will be useful for running automated regression tests on CI or CD tools, getting the results directly from these CI/CD tools, then transferring to other reporting systems.

Last but not least, relates to the connection between an application under test and test automation system, the last layer - test adaption provides the necessary code to adapt the automated test scripts for various components or interfaces of the application under test. It also provides different adaptors for connecting to the application under test via APIs, protocols, or other devices. A notable example here can be called TestOps. It includes the interface between test management and test adaptation layer which copes with the selection of appropriate adaptors in relation to the chosen test configuration.

To conclude, a test automation solution is a software system, thus it can be implemented like others. You can use any software engineering approach, from structured to software technologies and tools. Furthermore, team structure and programming skills are needed.

It is important to understand that a generic test automation architect has four layers: test generation, test definition, test execution, and test adaptation. You may not need to have all of them. For example, your team is just about to start introducing test automation, you should combine test execution and test adaptation in one layer.

In summary, this foundation course introduces various aspects of test automation that are fundamental to the initial stages of implementing automation into testing projects. After 4 chapters, you've learned about: test automation objectives, benefits, and what risks to expect; key considerations when applying automation, regarding organization and AUT environments; test automation pyramid - what it is and how to implement at different levels; and finally, the 4 layers of automation architecture.

Katalon Academy is working on even more courses to help you learn about test automation and immediately put them into practice with Katalon tools.

In the meantime, you can watch more of our existing courses by clicking on the Courses button right on the top menu bar.

Thank you, and happy learning!