Don’t let flaky tests undermine the value of your test automation Learn for free
This course covers everything you need to know about using Katalon Studio for test reporting, from keeping track of test executions, customizing and investigating test logs, to getting more advanced analytics from test results.
In four chapters, you’ll learn about:
00:05-00:34
Hi, and welcome to the first chapter where we're going to explore the Studio's execution log and reports at different execution levels, including test case, test suite, and test suite collection. We're also guiding you on how to customize the log viewer and set up video recordings for advanced troubleshooting and debugging. First off, let's create a sample project in Katalon Studio. If you haven't installed the tool yet, pause this video, go to Katalon.com/download, and install it.
00:35-01:00
Now, go to File > New Sample Project. We'll use the Shopping cart sample for this course. Wait for a few seconds and you're good to go. On the left sidebar, in the 'Test Cases' folder, we've already got some samples. The same thing goes for the 'Test Suites' folder below. You can see there are already some test suites and a test suite collection.
01:01-01:20
You can group relevant test cases into one suite for better management and execution and run it every time your source code has a new change. For example, you can put all the test cases for regression testing into a suite. A test suite allows you to categorize test cases in such a way that matches your planning and analysis needs.
01:21-02:10
At a higher level, we got test suite collections, which is symbolized by a file-with-a-dock icon. As its name suggests, it's a group of many test suites, which gives you more options to plan your test executions. Like in this one, it has 3 suites, and each one has one or more test cases.
We usually use test suite collections to run many test suites across multiple platforms. Just by double-clicking on this value here in the Run with column, you can assign or change the execution environment of a suite. Also, the good part about using a test suite collection is that you can run tests in parallel mode, instead of one after another (aka the Sequential mode), to reduce the execution time, especially on a large number of tests.
02:11-02:27
You can also have the option to create a dynamic test suite by right-clicking on the Test Suites folder. For the concept, a dynamic suite allows you to gather test cases automatically using search queries. But in this course, we don't need to use it. It will be discussed in another course for test management.
02:28-03:00
Moving on to the Reports folder - where we're going to see the reports of test suite and test suite collection executions. Since we haven't run any tests yet, there's nothing to see here. Last but not least, we'll learn about TestOps. It's a part of the Katalon platform that is used for test orchestration and analytics, from which you can get more actionable insights into test performance, trends, and the AUT's quality. That's all you need to know for now. In the next lessons, we'll run some tests, explore the Log Viewer and have a look at their reports. See you.
00:05-00:28
Welcome back! Now, time for some actions - we'll run some tests and learn how to work around the Execution Log Viewer in Katalon Studio. Simply click on the Run button on the menu bar to execute this test case with Chrome - as a default browser. You can also change it to other browsers in UI or headless mode by clicking on the dropdown icon on the side.
00:29-00:45
Alright, our test is up and running. In Studio, under the Log Viewer tab, every executed step is being displayed under a Tree View on the left pane. You can click on each step and see the detailed log messages on the right pane.
00:46-01:20
The Log Viewer is set up to always update you with real-time messages. It automatically scrolls down to show the latest executed steps by default. But if you want to stop this behavior, let's say, to look at or verify a particular message, you can select Scroll Lock - located in the top right corner here. Right beside is the Tree View toggle - the default display option of the Log Viewer. It gives a structural way to view how the test case or test suite is organized. You can switch it to the Tabular View by clicking on the toggle.
01:21-01:57
In this view mode, you can choose to filter the executed steps with these little toggles on the side. You can list out particular steps (passed, failed, error, warning, and not-run) according to your intention of viewing or troubleshooting. These filters are extremely useful when your tests have a lot of steps - which could give eye pain when scanning through to troubleshoot or investigate problems. I personally prefer the Tree View due to its structural view and the ability to collapse and expand to see the detailed messages.
01:58-02:34
To save time, let's skip to the part when our test is finished. Boop! Alright, back to the Tree View mode. As you can see, the test failed at step 5. You can select these up and down toggles on the side to show the previous or next failure. Or click on the toggle at the bottom to show only failed steps. The other two toggles at the top will allow you to collapse and expand all the executed steps. That's it for the side menu.
02:35-03:00
Moving up to this corner... We will ignore this eye symbol for now since it's for test suite collection. We'll get back to it later. Next, you can choose Word Wrap to make all the messages fit into the right pane so you can read them more easily.
03:01-03:20
Last but not least, you can minimize this whole section to have more working space to design your test cases... You can still see the Log Viewer or Console in a window by clicking on these options on the right-hand side. And to have a better view for troubleshooting or debugging, you can maximize the Log Viewer or Console to see all the activities and log messages more clearly.
02:28-03:38
Speaking of Console, you may wonder about the difference between this and the Log Viewer. Well, the Log Viewer shows only the report of the test execution itself, meanwhile the Console shows more of a complete picture - which displays all run-time activities while Katalon Studio runs the tests.
03:39-04:15
And in case you also wondered about the Event Log, it shows the activities of the plugins and integrations that are enabled for your test run and the runtime logs of Katalon Studio itself. Lastly, the Self-healing part - where all the broken object locators during test runs are listed out and can be quickly updated with just a few clicks. There's a separate course for this function. Make sure to check it out after finishing this course. That's all you need to know about the Log Viewer to keep track of your test runs, investigate and debug. In the next lesson, we're going to run some test suites in different modes and have a look at their reports in Katalon Studio.
00:05-00:32
Well done! You are at the first chapter's final lesson. We're going to run some test suites in different conditions and look at their reports. Let's start off with the CLI mode - command-line interface, shall we? To run Katalon tests via CLI commands in a local terminal, make sure you have Katalon Runtime Engine ready to use on your local machine. If you haven't, then pause the video, download, and install it. You can find the guideline easily by searching on Katalon Docs or Google.
00:33 -01:18
Now, get back to the Studio. To generate the CLI command:
- Click on the Command Builder button next to the Run button.
- As this dialog box appears, choose your test suite.
- Select your Organization, here I choose Katalon
- Click on the Generate Command button.
- Copy it to the clipboard.
Then, navigate to your Katalon Studio build folder, open a terminal window, paste the command in, and press enter. You will see the test suite begins to run. Now, some time-traveling.
01:19-01:48
Boop! And our test execution is finished. In Katalon Studio, click the Reports folder, find and choose our test suite execution. This suite has two test cases. And by looking at the green tick, you know that they both passed. In the middle is the summary section, including the overall summary of the suite execution (time, status, etc.) and the settings and the environment for that execution as well.
01:49-02:24
However, in the Log Viewer, there isn't any information about this suite execution. The same thing goes for the Console section. That's because the suite is actually executed on your local machine, not Katalon Studio. You can switch to your local terminal to see all the activities there, just like what you've seen in the Studio's Console earlier. But remember, all the information here will be gone for good once you close it. When building the command line, you may want to set it up to automatically upload these reports to Katalon TestOps and view them later. I'll show you how to do it later in this course.
02:25-03:08
Another way to view the execution report is through the Report folder in Katalon Studio. On the top right corner, click on Show Test Case Details to open the Test Case's Log section. Under the Test Log tab, all the executed steps and their status are shown for this selected test case. You can, again, filter out passed, failed, not-run steps, and so on by using these filtering toggles above. You can also search for particular steps by using the search bar. For example, I type in ''addtocart,'' step 4 appears as it contains this keyword.
03:09-03:40
At the bottom, the Log Information will give you more details about a particular test step, including its name, starting and ending execution time, duration, description, and the log message. Aside is the Log Image where you can see the screenshot of the AUT whenever the ''take screenshot'' keyword is used or an error occurs. And that's all about the reports of test suite execution in CLI mode on your local machine.
03:41-04:10
Now, let's move on to run a test suite collection directly in Katalon Studio. Will there be any difference regarding the report? Here, our test suite collection has 3 test suites. I'm going to run each one with a different browser and use the parallel execution mode, meaning the 3 suites will be executed simultaneously. These are the benefits of using a test suite collection as I've mentioned earlier.
04:11-04:50
So, looking at the Log Viewer, it starts to display the executed steps (it didn't show anything here when you ran tests in the console mode on your local machine). Right under the Log Viewer tab, you can see the number of executed tests, their status, and the visualized status bar of the execution. The browsers are being opened and the test steps are being executed automatically. At the bottom, looking at my Dock, you can see Microsoft Edge, Chrome, and then Firefox are opened at the same time. It indicates that our suites are being run simultaneously in parallel mode.
04:51-05:17
Back to the Studio, one thing to note about the Log Viewer is it can only show the execution details of one test suite at a time. If you want to see how other suites are performing, click on this eye-symbol icon, choose the one you want to view and the Log Viewer will display accordingly. Well, all of our suites are up and running. Let's skip it to when they're all finished.
05:18-05:44
Wham! All done. Looks like there are some failed tests. You can see here 7/7 test cases are executed. To see the report, click on the Result tab. Here are all the overall details of the 3 test suite executions. In the Failed Test / Total column, it says that our first suite has 1 failed test out of the total 4. Meanwhile, the other two are both good.
05:45-06:04
Click on Show details to have a closer look at each test suite. This one has only one test case whose status is passed. Further click on Show test case details to see all the executed steps. You can also search for particular steps to view, which is quite convenient in case your test case has a lot of steps.
06:05-06:36
Alright, let's check the others... This last suite is actually the one that we've run on local machines earlier. All of its test cases passed so let's move on to investigate the other one, which actually has one failed test. For the context, this suite is applied data-driven testing... So, there are 4 iterations here... This second one failed at step 5. You can expand to view all the activities taking place in this step. Hmmmm, let's see.
06:37-07:04
It said here, ''unable to set text of the object.'' You can see more log messages about this in the Information section below. It could be due to flakiness that the object could not be found, since all other tests in this suite passed, given the same test steps and objects. Or maybe the application does not function well with this particular test data. That's my guess on the root cause. Moving on.
07:05-07:33
Another way to quickly troubleshoot the problem is going through the Log Viewer. Expand the failed test case, select the failed step. You can see in the right pane, it said ''Root Cause'' - ''unable to set text of the object.'' Click on the address to quickly navigate to the object's settings. From here, you can start investigating. Let's say the locator was incorrect, you can quickly fix it and then re-run this test case.
07:34-07:53
Well done, you've finished the first chapter. Now, you know how to work around the execution log and test reports in Katalon Studio. In the next chapter, we're going to learn how to customize the test log and set up video recordings for troubleshooting and debugging. See you then.
m
good
Good
Good
30-DAY PREMIUM TRIAL