Chapter 3 - Exporting Test Reports in Katalon Studio
Chapter 4 - TestOps Integration for Advanced Test Reporting
This course covers everything you need to know about using Katalon Studio for test reporting, from keeping track of test executions, customizing and investigating test logs, to getting more advanced analytics from test results.
In four chapters, you’ll learn about:
Chapter 1 - Working With Execution Log and Reports in Katalon Studio
Hi, and welcome to the first chapter where we're going to explore the Studio's execution log and reports at different execution levels, including test case, test suite, and test suite collection. We're also guiding you on how to customize the log viewer and set up video recordings for advanced troubleshooting and debugging. First off, let's create a sample project in Katalon Studio. If you haven't installed the tool yet, pause this video, go to Katalon.com/download, and install it.
Now, go to File > New Sample Project. We'll use the Shopping cart sample for this course. Wait for a few seconds and you're good to go. On the left sidebar, in the 'Test Cases' folder, we've already got some samples. The same thing goes for the 'Test Suites' folder below. You can see there are already some test suites and a test suite collection.
You can group relevant test cases into one suite for better management and execution and run it every time your source code has a new change. For example, you can put all the test cases for regression testing into a suite. A test suite allows you to categorize test cases in such a way that matches your planning and analysis needs.
At a higher level, we got test suite collections, which is symbolized by a file-with-a-dock icon. As its name suggests, it's a group of many test suites, which gives you more options to plan your test executions. Like in this one, it has 3 suites, and each one has one or more test cases.
We usually use test suite collections to run many test suites across multiple platforms. Just by double-clicking on this value here in the Run with column, you can assign or change the execution environment of a suite. Also, the good part about using a test suite collection is that you can run tests in parallel mode, instead of one after another (aka the Sequential mode), to reduce the execution time, especially on a large number of tests.
You can also have the option to create a dynamic test suite by right-clicking on the Test Suites folder. For the concept, a dynamic suite allows you to gather test cases automatically using search queries. But in this course, we don't need to use it. It will be discussed in another course for test management.
Moving on to the Reports folder - where we're going to see the reports of test suite and test suite collection executions. Since we haven't run any tests yet, there's nothing to see here. Last but not least, we'll learn about TestOps. It's a part of the Katalon platform that is used for test orchestration and analytics, from which you can get more actionable insights into test performance, trends, and the AUT's quality. That's all you need to know for now. In the next lessons, we'll run some tests, explore the Log Viewer and have a look at their reports. See you.
Welcome back! Now, time for some actions - we'll run some tests and learn how to work around the Execution Log Viewer in Katalon Studio. Simply click on the Run button on the menu bar to execute this test case with Chrome - as a default browser. You can also change it to other browsers in UI or headless mode by clicking on the dropdown icon on the side.
Alright, our test is up and running. In Studio, under the Log Viewer tab, every executed step is being displayed under a Tree View on the left pane. You can click on each step and see the detailed log messages on the right pane.
The Log Viewer is set up to always update you with real-time messages. It automatically scrolls down to show the latest executed steps by default. But if you want to stop this behavior, let's say, to look at or verify a particular message, you can select Scroll Lock - located in the top right corner here. Right beside is the Tree View toggle - the default display option of the Log Viewer. It gives a structural way to view how the test case or test suite is organized. You can switch it to the Tabular View by clicking on the toggle.
In this view mode, you can choose to filter the executed steps with these little toggles on the side. You can list out particular steps (passed, failed, error, warning, and not-run) according to your intention of viewing or troubleshooting. These filters are extremely useful when your tests have a lot of steps - which could give eye pain when scanning through to troubleshoot or investigate problems. I personally prefer the Tree View due to its structural view and the ability to collapse and expand to see the detailed messages.
To save time, let's skip to the part when our test is finished. Boop! Alright, back to the Tree View mode. As you can see, the test failed at step 5. You can select these up and down toggles on the side to show the previous or next failure. Or click on the toggle at the bottom to show only failed steps. The other two toggles at the top will allow you to collapse and expand all the executed steps. That's it for the side menu.
Moving up to this corner... We will ignore this eye symbol for now since it's for test suite collection. We'll get back to it later. Next, you can choose Word Wrap to make all the messages fit into the right pane so you can read them more easily.
Last but not least, you can minimize this whole section to have more working space to design your test cases... You can still see the Log Viewer or Console in a window by clicking on these options on the right-hand side. And to have a better view for troubleshooting or debugging, you can maximize the Log Viewer or Console to see all the activities and log messages more clearly.
Speaking of Console, you may wonder about the difference between this and the Log Viewer. Well, the Log Viewer shows only the report of the test execution itself, meanwhile the Console shows more of a complete picture - which displays all run-time activities while Katalon Studio runs the tests.
And in case you also wondered about the Event Log, it shows the activities of the plugins and integrations that are enabled for your test run and the runtime logs of Katalon Studio itself. Lastly, the Self-healing part - where all the broken object locators during test runs are listed out and can be quickly updated with just a few clicks. There's a separate course for this function. Make sure to check it out after finishing this course. That's all you need to know about the Log Viewer to keep track of your test runs, investigate and debug. In the next lesson, we're going to run some test suites in different modes and have a look at their reports in Katalon Studio.
Well done! You are at the first chapter's final lesson. We're going to run some test suites in different conditions and look at their reports. Let's start off with the CLI mode - command-line interface, shall we? To run Katalon tests via CLI commands in a local terminal, make sure you have Katalon Runtime Engine ready to use on your local machine. If you haven't, then pause the video, download, and install it. You can find the guideline easily by searching on Katalon Docs or Google.
Now, get back to the Studio. To generate the CLI command:
- Click on the Command Builder button next to the Run button.
- As this dialog box appears, choose your test suite.
- Select your Organization, here I choose Katalon
- Click on the Generate Command button.
- Copy it to the clipboard.
Then, navigate to your Katalon Studio build folder, open a terminal window, paste the command in, and press enter. You will see the test suite begins to run. Now, some time-traveling.
Boop! And our test execution is finished. In Katalon Studio, click the Reports folder, find and choose our test suite execution. This suite has two test cases. And by looking at the green tick, you know that they both passed. In the middle is the summary section, including the overall summary of the suite execution (time, status, etc.) and the settings and the environment for that execution as well.
However, in the Log Viewer, there isn't any information about this suite execution. The same thing goes for the Console section. That's because the suite is actually executed on your local machine, not Katalon Studio. You can switch to your local terminal to see all the activities there, just like what you've seen in the Studio's Console earlier. But remember, all the information here will be gone for good once you close it. When building the command line, you may want to set it up to automatically upload these reports to Katalon TestOps and view them later. I'll show you how to do it later in this course.
Another way to view the execution report is through the Report folder in Katalon Studio. On the top right corner, click on Show Test Case Details to open the Test Case's Log section. Under the Test Log tab, all the executed steps and their status are shown for this selected test case. You can, again, filter out passed, failed, not-run steps, and so on by using these filtering toggles above. You can also search for particular steps by using the search bar. For example, I type in ''addtocart,'' step 4 appears as it contains this keyword.
At the bottom, the Log Information will give you more details about a particular test step, including its name, starting and ending execution time, duration, description, and the log message. Aside is the Log Image where you can see the screenshot of the AUT whenever the ''take screenshot'' keyword is used or an error occurs. And that's all about the reports of test suite execution in CLI mode on your local machine.
Now, let's move on to run a test suite collection directly in Katalon Studio. Will there be any difference regarding the report? Here, our test suite collection has 3 test suites. I'm going to run each one with a different browser and use the parallel execution mode, meaning the 3 suites will be executed simultaneously. These are the benefits of using a test suite collection as I've mentioned earlier.
So, looking at the Log Viewer, it starts to display the executed steps (it didn't show anything here when you ran tests in the console mode on your local machine). Right under the Log Viewer tab, you can see the number of executed tests, their status, and the visualized status bar of the execution. The browsers are being opened and the test steps are being executed automatically. At the bottom, looking at my Dock, you can see Microsoft Edge, Chrome, and then Firefox are opened at the same time. It indicates that our suites are being run simultaneously in parallel mode.
Back to the Studio, one thing to note about the Log Viewer is it can only show the execution details of one test suite at a time. If you want to see how other suites are performing, click on this eye-symbol icon, choose the one you want to view and the Log Viewer will display accordingly. Well, all of our suites are up and running. Let's skip it to when they're all finished.
Wham! All done. Looks like there are some failed tests. You can see here 7/7 test cases are executed. To see the report, click on the Result tab. Here are all the overall details of the 3 test suite executions. In the Failed Test / Total column, it says that our first suite has 1 failed test out of the total 4. Meanwhile, the other two are both good.
Click on Show details to have a closer look at each test suite. This one has only one test case whose status is passed. Further click on Show test case details to see all the executed steps. You can also search for particular steps to view, which is quite convenient in case your test case has a lot of steps.
Alright, let's check the others... This last suite is actually the one that we've run on local machines earlier. All of its test cases passed so let's move on to investigate the other one, which actually has one failed test. For the context, this suite is applied data-driven testing... So, there are 4 iterations here... This second one failed at step 5. You can expand to view all the activities taking place in this step. Hmmmm, let's see.
It said here, ''unable to set text of the object.'' You can see more log messages about this in the Information section below. It could be due to flakiness that the object could not be found, since all other tests in this suite passed, given the same test steps and objects. Or maybe the application does not function well with this particular test data. That's my guess on the root cause. Moving on.
Another way to quickly troubleshoot the problem is going through the Log Viewer. Expand the failed test case, select the failed step. You can see in the right pane, it said ''Root Cause'' - ''unable to set text of the object.'' Click on the address to quickly navigate to the object's settings. From here, you can start investigating. Let's say the locator was incorrect, you can quickly fix it and then re-run this test case.
Well done, you've finished the first chapter. Now, you know how to work around the execution log and test reports in Katalon Studio. In the next chapter, we're going to learn how to customize the test log and set up video recordings for troubleshooting and debugging. See you then.
Chapter 2 - Advanced Settings for Troubleshooting Failed Tests
Hi and welcome back! In this lesson, I will show you how to customize the execution log to further investigate failed tests and debug. When a test is executed in Katalon Studio, the Log Viewer and Console show only necessary information at certain logging levels to keep everything neat and clean. Looking at this, the Console only displays certain details at INFO and DEBUG logging levels.
However, in some cases, you need more information to investigate, troubleshoot or debug your failed tests. Or sometimes you want to lessen your log messages to save resources or reduce the execution time when running a large number of tests. Go to the Include folder > config > log.properties. This is the file where you can customize the logging levels in Katalon Studio.
And to demonstrate, I will use the TRACE level. It is the deepest level of logs, providing the details of all the activities taking place from both the application and Katalon Studio - even more details than the DEBUG level used by default in Studio.
Some other common logging levels you may hear of include:
- Fatal: for very severe error events
- Error: for error events that prevent normal program execution, but might still allow the application to continue running
- Warn: for potentially harmful events to end-users or system managers
- Info: the standard level showing purely informative messages on the progress of the application
- Debug: showing the details used by developers to troubleshoot when running the application in test environments
What logging levels to use really depends on your intention. Let's say, if you want to lessen the log messages, you can add a simple INFO-level command line (like logging.level.testcase=INFO) to this log.properties file.
Back to our TRACE logging line, which has already been added here. There's a hashtag #, indicating that it is currently disabled. Simply delete it to enable the command, save, and then re-run the test case. While the test is running, let's switch the Console to see what's happening. So, all the activities are being listed out, even including the details of the Smart Wait function being triggered, the behaviors of finding an object, clicking on that object, confirming that object is being clicked on, checking object, checking timeout, etc.
The TRACE logging level is beneficial when you need a complete detailed process of test execution to investigate the causes and debug. Now, how about the Log Viewer? It's also affected by the TRACE command like the Console. In each executed step, all the relevant activities are listed for you to view and troubleshoot your test. That's it for this lesson. In the next one, you'll learn how to set up video recording for reporting.
Hi, welcome to the second half of this chapter - where we're guiding you on setting up video recording for web UI testing. Video recording is useful when you need to rewatch your failed tests, especially when they are executed in headless mode.
You will have to install a 3rd-party library (FFmpeg) as a precondition for video encoding on your local machine or the server where the tests are performed. Pause this video, hop on to Katalon Docs, search for Record browser-based videos page, scroll down to the Install FFmpeg library section and follow the instructions.
Once you've done that, in Katalon Studio:
- Go to Projects > Settings > Execution
- Check on Record video during execution
- In this case, I want video recordings for headless execution mode. So I'll select Browser-based Recorder since it supports headless mode
Next, video recordings for all test case executions or just for failed tests? If you run a large number of tests, for example 1,000 test cases, the second option is better. You rarely watch the records of passed tests, right? And remember, video recordings may slow down your execution time. That said, I choose All test cases here, just for the purpose of demonstration.
Next, we'll need to set up the window size for our browser recording (for a better video resolution). In the same Project Setting dialog box:
- Choose Desired Capabilities > Web UI > Chrome (headless)
- Select Add
- Add value to the last column
- Type ''--window-size=1500,1000'' in the Value column
- All good! Click on Apply and Close.
Great! Now that we've gone through all the required settings, let's see how it works by running a test suite (since this feature supports test suites and test suite collections). Let's go for Chrome (headless). In this execution mode, it won't open the browser UI and you cannot see the test runs on the screen. This means fewer resources and faster execution time faster compared to the UI mode - which is very beneficial when you run a large number of tests or when running tests in CI pipelines.
Although nothing's shown on screen, you can still see the execution details are still being logged inside the Log Viewer and Console, helping keep track of the test runs. Now, it's time for some time traveling.
Wham! It's done. ''Video recording for this test case.'' Looks like everything's good. Click on Result, and hit on the small Play icon in the Video column to open the recording. Press Play and you can re-watch all the steps, but this time, being executed on the browser UI, like you normally see when running tests in the UI mode. This will help you better at detecting what went wrong at which step. Very cool, right?
That's it! That's the end of our chapter. You've learned how to customize the execution log at different logging levels and generate video reports for web UI testing in Katalon Studio. In the next chapters, we're going to export and walk through Studio's test reports in different formats, set up automatic notification emails, and use Katalon TestOps to get more insights into your test results.
Chapter 3 - Exporting Test Reports in Katalon Studio
Welcome back. In this chapter, we'll walk you through how to export and interpret test reports in HTML, CSV, and PDF. You'll also learn the configuration for sending test reports via email automatically. When a test suite is executed, click on the Result tab here, or you can go to Reports, then:
- Choose the test suite
- Select Export report
- Let's go for HTML first
- Select the location to save your report
- Finally, click Open
By default, the HTML report will be automatically opened on your browser. You can see there are 2 main parts. The first one is the summary of the execution environment, including the hostname, operating system, Katalon version, and browsers. For the second part, at a higher level, it's the test suite execution summary - including the name, runtime, and final status... There are two tests here and both passed.
Below, it displays all the information of each test, including their summary and test logs - which are basically the execution details you're seeing here in the Test Case's log section in Katalon Studio. Similarly, you can export an HTML report for a test suite collection by following the same steps. Select Export Report > HTML > Choose a place to save it > Then Open.
It will open the report index, displaying the overall results of the executed test suites. Select any suite to view its report - which is exactly the same as the one we've just gone through earlier, regarding the layout. Looking at this suite, for instance, it has 4 test cases - 3 passed and 1 failed. Expand and you can see the root cause along with the test log message at the top.
Or you can scroll down to the failed step and investigate the root cause directly from there. Seems like it failed because it was ''unable to set text of object...'' At the bottom, Katalon Studio also captured a screenshot at the failed step, giving you more insights into the problem. On your local PC, you can see there are four exported HTML files, including the index and specific reports for each test suite execution. Simply convert the whole folder into a zip file and send it to your team members or managers via email to report on your testing process.
Other than HTML, you can also export reports in CSV and PDF formats. Choose a test suite report, Export Report, here I go for CSV (Details) since it has more information to look at, click Open. You can view the CSV file using MS Excel or any IDE like VS Code.
It provides a very simple report. In the second line, you can see the overall information of the execution, including the browser environment, description, tag, start and end time, duration, and most importantly the status. Here it says, ''FAILED,'' indicating that there's at least one test failed. Below, there are four sections - matching the four executed tests, each separated from one another by a blank row.
The first line of each section is the test case name and below are the executed steps. Looking at the Status column, you can quickly spot that this second test failed at the Checkout step. That's pretty much it for the CSV (details) format. Let's move on and export a PDF report for this very same test suite.
Here it is. The layout is quite similar to the HTML format, right? The headline is the test suite name, then there are environment details and the summary of the test suite execution, including the list of test cases and their status. The next pages are where you can see the particular report of each test, including the test case name, its execution status, the main details of each test step and its status.
Similar to the CSV report, the second test is reported as ''FAILED.'' Scrolling down... you can see it failed at step 5 - ''unable to set text of object'' - along with a screenshot of the browser. The two other tests in this suite are both good - both passed. What about the test suite collection? Can you export CSV or PDF reports? Well, the answer is NO since HTML is the only option. And that's the end of this lesson. In the next one, we're going to show you how to send Katalon test reports via email.
One of the major benefits of test automation is that all the test steps run automatically, allowing you to go elsewhere and handle other tasks in the meantime. Here's the scenario: Oftentimes, you want to get back for the reports as soon as the execution completes. Or your teams normally are making night builds and you want the test results to be in everyone's mailboxes the next morning. In those cases, having notification and receiving test reports automatically is super helpful. With Katalon, you can do exactly that by configuring it to send out test reports automatically via email when test runs are finished.
On the menu bar, choose Project > Settings > then Email. First, you need to define your mail server that Katalon Studio will use for sending reports. In this course, I'm using my Gmail as the server, here's what I need to fill in:
- ''smtp.gmail.com'' for host
- The port is ''465''
- The credentials of my Gmail account
- Finally, ''SSL'' for Protocol
If you're using other servers like Yahoo or Outlook, go to Katalon Docs >> Search and go to Project Settings page >> and follow the instructions in the Email Settings sections.
Note that, if your account is set up with two-step authentication, you have to create an App password and use it for these email settings. Otherwise, the configuration would not be successful. Go and search ''Sign in with App Passwords'' on Google. Follow the instructions on the Google Support page, and you'll be good to go. Another way is to turn off the two-step authentication of your email account, but we do not recommend this at all.
So, after filling in all required information, to check whether the settings work, enter your email address as Recipients and press Send a test email. If it announces ''Test email sent successfully'' then it's all good. You can also attach the reports to the email by checking this box and selecting the formats you want.
By default, Katalon Studio is set up to send ALL email reports for Test Suite executions, including the Test Suites in a Test Suite Collection. To keep your mailbox tidy, you can go for the option of sending emails for:
- Only test suite executions that have failed test cases, or
- Skip sending email reports for individual Test Suites in the Test Suite Collection.
In the Template section, you can edit the email content... Now, we'll run a test suite and see how it works. And let's skip it when our execution is finished. Alright, it's done! Lemme go ahead and check my mailbox.
Ah, our mail has arrived. You can see it's sent by my account and the mail subject is the same as what we've set up. The email will give you a summary of the test execution. Two passed, none failed. There's also an attached file where you can find and download reports for the execution details.
That's all for the email reports. You can add your team members to the Recipients list in the email settings, CC to your managers, and BCC to your client to inform everyone of your testing process. In this chapter, you've learned how to export reports in different formats and set up automatic email reports as well. Moving on to the next one, we'll show you how to utilize Katalon TestOps to make the most use of your test reporting.
Chapter 4 - TestOps Integration for Advanced Test Reporting
Congrats! You've made it to the last chapter of this course - where we're going to show you how the integration of Katalon Studio and TestOps can help you manage and get more actionable insights from your test results and performance.
Katalon TestOps is a platform that allows you to orchestrate tests and DevOps effectively. But in this course, we'll focus on its advanced reporting capabilities, from which you and teammates can get the full picture of your test performance, passed-failed execution trends, and the quality of your application under test.
Under the Test Explore tab, select TestOps > Test Runs. If it doesn't show any test report, but instead the line ''Please check TestOps integration,'' chances are you haven't integrated Katalon Studio with TestOps.
To set up the integration, simply click on the TestOps icon on the tool bar or go to Project > Settings > Katalon TestOps. Your information will be auto-filled for authentication. Check on ''Enable Katalon TestOps integration.'' Then, choose your organization... team... and project - where you want to save your test results on TestOps. Then Apply and Close.
You can also create new environments or organizations for the integration by following these steps:
- Login to TestOps - testops.katalon.io - using your Katalon account. You can create one for free if you don't have any
- You can see this is my current organization with recent projects and the latest runs
- To create a new org, click on the dropdown icon on the menu bar
- Then, select Create new organization
- Give it a name, keep it all as default and finish
Back to Katalon Studio. Refresh this Test Runs tab to retrieve all the test reports that have been uploaded to TestOps for the selected project. Once the integration is enabled, all the test runs in Katalon Studio will be uploaded to TestOps automatically. However, the ones BEFORE the enabled integration are NOT.
That means this test suite result - I ran before the enabled integration - hasn't been uploaded to TestOps yet. To do so, manually select Katalon TestOps > Upload > Choose Team and Project > Then Upload. Wait for a few seconds... Refresh to fetch the change... Ah, our test result has been successfully uploaded - ID 255. Let's click on the ID to navigate and view the execution details directly on TestOps.
Here's the summary report for our test run #255... Since our tests passed, there's nothing to see in the Failed Test Results. Scrolling down to the Test Suite section, click on it and you can see the historical executions of this particular suite. Click on Test Results, it will show you all the executions in this test run. You can search for particular tests or filter them by their status, test suites, time/duration, and profile.
For each test, it lists out all the executed steps as you're seeing here in the Logs section. You can switch to Text view or remain in the Tree view mode, just like when working with the Log Viewer in Katalon Studio... There's also a download button in case you want to save it in your PC or send it to others. Right above is the Failed Assertions where it would explain why the test failed. But since ours passed, ''there's no data to display.''
Alright, back to the Studio, let's run a failed test suite this time... And again, to save our time, skip it to the end of the execution. You can see here in the Job Progress, our test report is being uploaded to TestOps automatically. The same goes for every other test suite execution since the Studio-and-TestOps integration is enabled, as I've mentioned earlier. It's actually done now! Let's head to TestOps and see how things are going there.
So, in TestOps' Reports category, our report has been uploaded successfully - ID 256... 1 passed, and 1 failed test. In this Test Runs tab, you can see the summary details of the failed test result. Click on it and you can further investigate the root cause with Error Message, Screenshot Attachments, and Test Logs. Another cool thing about using TestOps' reporting is that you can link, report bugs on Jira, and even assign a maintainer - who will fix this - directly from here. We'll discuss these in detail for another course.
Now, let's move on exploring TestOps a little bit more... You can go to Dashboard and view all the recent test activities for a particular project - this week or the week before. The TestOps team is working on rolling out new features and enhancements soon, so stay tuned.
Continue! Looking at this week, we have 2 test runs, 2 test cases, so 4 test results. Based on the charts and colors, we can tell there were:
- 3 passed and 1 failed test results
- And equally 1 passed and 1 failed test runs
Clicking on View All on the bottom right corner, you'll go to the Reports category - where you can view all the test runs for this particular project. Let's quickly walk through this together, shall we?
There is a chart here summarizing the pass/fail trend of your most recent test runs, dated back to 28 days max. Scrolling down, you can see all historical test run data, up-to-date. The first one is the latest test run that we went through just a moment ago. And in case you're wondering what these letters stand for:
- P means Passed
- F - Failed
- E - Error
- And I for Incomplete
Moving on to Summary by Test Run Results... here you see the overall status and performance of your test cases. This chart gives you insights on the trends of your test executions over time, how many test results and test cases passed or failed, along with their time duration. And below is the summary of test runs weekly overtime. Through all of these data, you can know more about the performance and quality of the application under test as well as the automated tests themselves - like how how flaky they are and the execution trends.
Well done! That's the end of our course. To recap, here's what you've learned:
- How to work with Log Viewer and Reports in Katalon Studio
- Exporting and understanding reports in different formats like CVS, HTML, and PDF
- Setting up to send test report automatically via email
- Finally, utilizing TestOps' advanced reporting capabilities to centralize all test execution data and get more insights on your test performance over time
So, what's next? Practice makes perfect! Go to Download on Katalon.com, install Studio, and start applying what you've learned in this course into your project. Katalon Academy is working on even more courses to help you learn about test automation and immediately put them into practice with Katalon tools. In the meantime, you can watch more of our existing courses by clicking on the Courses button right on the top menu bar. Thank you, and happy learning!
NEXT COURSES TO LEARN