Katalon Studio: How To Work With Execution Logs and Test Reports

Hi and welcome back! In this lesson, I will show you how to customize the execution log to further investigate failed tests and debug. When a test is executed in Katalon Studio, the Log Viewer and Console show only necessary information at certain logging levels to keep everything neat and clean. Looking at this, the Console only displays certain details at INFO and DEBUG logging levels.

However, in some cases, you need more information to investigate, troubleshoot or debug your failed tests. Or sometimes you want to lessen your log messages to save resources or reduce the execution time when running a large number of tests. Go to the Include folder > config > log.properties. This is the file where you can customize the logging levels in Katalon Studio.

And to demonstrate, I will use the TRACE level. It is the deepest level of logs, providing the details of all the activities taking place from both the application and Katalon Studio - even more details than the DEBUG level used by default in Studio.

Some other common logging levels you may hear of include:

- Fatal: for very severe error events

- Error: for error events that prevent normal program execution, but might still allow the application to continue running

- Warn: for potentially harmful events to end-users or system managers

- Info: the standard level showing purely informative messages on the progress of the application

- Debug: showing the details used by developers to troubleshoot when running the application in test environments

What logging levels to use really depends on your intention. Let's say, if you want to lessen the log messages, you can add a simple INFO-level command line (like logging.level.testcase=INFO) to this log.properties file.

Back to our TRACE logging line, which has already been added here. There's a hashtag #, indicating that it is currently disabled. Simply delete it to enable the command, save, and then re-run the test case. While the test is running, let's switch the Console to see what's happening. So, all the activities are being listed out, even including the details of the Smart Wait function being triggered, the behaviors of finding an object, clicking on that object, confirming that object is being clicked on, checking object, checking timeout, etc.

The TRACE logging level is beneficial when you need a complete detailed process of test execution to investigate the causes and debug. Now, how about the Log Viewer? It's also affected by the TRACE command like the Console. In each executed step, all the relevant activities are listed for you to view and troubleshoot your test. That's it for this lesson. In the next one, you'll learn how to set up video recording for reporting.

Hi, welcome to the second half of this chapter - where we're guiding you on setting up video recording for web UI testing. Video recording is useful when you need to rewatch your failed tests, especially when they are executed in headless mode.

You will have to install a 3rd-party library (FFmpeg) as a precondition for video encoding on your local machine or the server where the tests are performed. Pause this video, hop on to Katalon Docs, search for Record browser-based videos page, scroll down to the Install FFmpeg library section and follow the instructions.

Once you've done that, in Katalon Studio:

- Go to Projects > Settings > Execution

- Check on Record video during execution

- In this case, I want video recordings for headless execution mode. So I'll select Browser-based Recorder since it supports headless mode

Next, video recordings for all test case executions or just for failed tests? If you run a large number of tests, for example 1,000 test cases, the second option is better. You rarely watch the records of passed tests, right? And remember, video recordings may slow down your execution time. That said, I choose All test cases here, just for the purpose of demonstration.

Next, we'll need to set up the window size for our browser recording (for a better video resolution). In the same Project Setting dialog box:

- Choose Desired Capabilities > Web UI > Chrome (headless)

- Select Add

- Add value to the last column

- Type ''--window-size=1500,1000'' in the Value column

- All good! Click on Apply and Close.

Great! Now that we've gone through all the required settings, let's see how it works by running a test suite (since this feature supports test suites and test suite collections). Let's go for Chrome (headless). In this execution mode, it won't open the browser UI and you cannot see the test runs on the screen. This means fewer resources and faster execution time faster compared to the UI mode - which is very beneficial when you run a large number of tests or when running tests in CI pipelines.

Although nothing's shown on screen, you can still see the execution details are still being logged inside the Log Viewer and Console, helping keep track of the test runs. Now, it's time for some time traveling.

Wham! It's done. ''Video recording for this test case.'' Looks like everything's good. Click on Result, and hit on the small Play icon in the Video column to open the recording. Press Play and you can re-watch all the steps, but this time, being executed on the browser UI, like you normally see when running tests in the UI mode. This will help you better at detecting what went wrong at which step. Very cool, right?

That's it! That's the end of our chapter. You've learned how to customize the execution log at different logging levels and generate video reports for web UI testing in Katalon Studio. In the next chapters, we're going to export and walk through Studio's test reports in different formats, set up automatic notification emails, and use Katalon TestOps to get more insights into your test results.

Welcome back. In this chapter, we'll walk you through how to export and interpret test reports in HTML, CSV, and PDF. You'll also learn the configuration for sending test reports via email automatically. When a test suite is executed, click on the Result tab here, or you can go to Reports, then:

- Choose the test suite

- Select Export report

- Let's go for HTML first

- Select the location to save your report

- Finally, click Open

By default, the HTML report will be automatically opened on your browser. You can see there are 2 main parts. The first one is the summary of the execution environment, including the hostname, operating system, Katalon version, and browsers. For the second part, at a higher level, it's the test suite execution summary - including the name, runtime, and final status... There are two tests here and both passed.

Below, it displays all the information of each test, including their summary and test logs - which are basically the execution details you're seeing here in the Test Case's log section in Katalon Studio. Similarly, you can export an HTML report for a test suite collection by following the same steps. Select Export Report > HTML > Choose a place to save it > Then Open.

It will open the report index, displaying the overall results of the executed test suites. Select any suite to view its report - which is exactly the same as the one we've just gone through earlier, regarding the layout. Looking at this suite, for instance, it has 4 test cases - 3 passed and 1 failed. Expand and you can see the root cause along with the test log message at the top.

Or you can scroll down to the failed step and investigate the root cause directly from there. Seems like it failed because it was ''unable to set text of object...'' At the bottom, Katalon Studio also captured a screenshot at the failed step, giving you more insights into the problem. On your local PC, you can see there are four exported HTML files, including the index and specific reports for each test suite execution. Simply convert the whole folder into a zip file and send it to your team members or managers via email to report on your testing process.

Other than HTML, you can also export reports in CSV and PDF formats. Choose a test suite report, Export Report, here I go for CSV (Details) since it has more information to look at, click Open. You can view the CSV file using MS Excel or any IDE like VS Code.

It provides a very simple report. In the second line, you can see the overall information of the execution, including the browser environment, description, tag, start and end time, duration, and most importantly the status. Here it says, ''FAILED,'' indicating that there's at least one test failed. Below, there are four sections - matching the four executed tests, each separated from one another by a blank row.

The first line of each section is the test case name and below are the executed steps. Looking at the Status column, you can quickly spot that this second test failed at the Checkout step. That's pretty much it for the CSV (details) format. Let's move on and export a PDF report for this very same test suite.

Here it is. The layout is quite similar to the HTML format, right? The headline is the test suite name, then there are environment details and the summary of the test suite execution, including the list of test cases and their status. The next pages are where you can see the particular report of each test, including the test case name, its execution status, the main details of each test step and its status.

Similar to the CSV report, the second test is reported as ''FAILED.'' Scrolling down... you can see it failed at step 5 - ''unable to set text of object'' - along with a screenshot of the browser. The two other tests in this suite are both good - both passed. What about the test suite collection? Can you export CSV or PDF reports? Well, the answer is NO since HTML is the only option. And that's the end of this lesson. In the next one, we're going to show you how to send Katalon test reports via email.

One of the major benefits of test automation is that all the test steps run automatically, allowing you to go elsewhere and handle other tasks in the meantime. Here's the scenario: Oftentimes, you want to get back for the reports as soon as the execution completes. Or your teams normally are making night builds and you want the test results to be in everyone's mailboxes the next morning. In those cases, having notification and receiving test reports automatically is super helpful. With Katalon, you can do exactly that by configuring it to send out test reports automatically via email when test runs are finished.

On the menu bar, choose Project > Settings > then Email. First, you need to define your mail server that Katalon Studio will use for sending reports. In this course, I'm using my Gmail as the server, here's what I need to fill in:

- ''smtp.gmail.com'' for host

- The port is ''465''

- The credentials of my Gmail account

- Finally, ''SSL'' for Protocol

If you're using other servers like Yahoo or Outlook, go to Katalon Docs >> Search and go to Project Settings page >> and follow the instructions in the Email Settings sections.

Note that, if your account is set up with two-step authentication, you have to create an App password and use it for these email settings. Otherwise, the configuration would not be successful. Go and search ''Sign in with App Passwords'' on Google. Follow the instructions on the Google Support page, and you'll be good to go. Another way is to turn off the two-step authentication of your email account, but we do not recommend this at all.

So, after filling in all required information, to check whether the settings work, enter your email address as Recipients and press Send a test email. If it announces ''Test email sent successfully'' then it's all good. You can also attach the reports to the email by checking this box and selecting the formats you want.

By default, Katalon Studio is set up to send ALL email reports for Test Suite executions, including the Test Suites in a Test Suite Collection. To keep your mailbox tidy, you can go for the option of sending emails for:

- Only test suite executions that have failed test cases, or

- Skip sending email reports for individual Test Suites in the Test Suite Collection.

In the Template section, you can edit the email content... Now, we'll run a test suite and see how it works. And let's skip it when our execution is finished. Alright, it's done! Lemme go ahead and check my mailbox.

Ah, our mail has arrived. You can see it's sent by my account and the mail subject is the same as what we've set up. The email will give you a summary of the test execution. Two passed, none failed. There's also an attached file where you can find and download reports for the execution details.

That's all for the email reports. You can add your team members to the Recipients list in the email settings, CC to your managers, and BCC to your client to inform everyone of your testing process. In this chapter, you've learned how to export reports in different formats and set up automatic email reports as well. Moving on to the next one, we'll show you how to utilize Katalon TestOps to make the most use of your test reporting.

Congrats! You've made it to the last chapter of this course - where we're going to show you how the integration of Katalon Studio and TestOps can help you manage and get more actionable insights from your test results and performance.

Katalon TestOps is a platform that allows you to orchestrate tests and DevOps effectively. But in this course, we'll focus on its advanced reporting capabilities, from which you and teammates can get the full picture of your test performance, passed-failed execution trends, and the quality of your application under test.

Under the Test Explore tab, select TestOps > Test Runs. If it doesn't show any test report, but instead the line ''Please check TestOps integration,'' chances are you haven't integrated Katalon Studio with TestOps.

To set up the integration, simply click on the TestOps icon on the tool bar or go to Project > Settings > Katalon TestOps. Your information will be auto-filled for authentication. Check on ''Enable Katalon TestOps integration.'' Then, choose your organization... team... and project - where you want to save your test results on TestOps. Then Apply and Close.

You can also create new environments or organizations for the integration by following these steps:

- Login to TestOps - testops.katalon.io - using your Katalon account. You can create one for free if you don't have any

- You can see this is my current organization with recent projects and the latest runs

- To create a new org, click on the dropdown icon on the menu bar

- Then, select Create new organization

- Give it a name, keep it all as default and finish

Back to Katalon Studio. Refresh this Test Runs tab to retrieve all the test reports that have been uploaded to TestOps for the selected project. Once the integration is enabled, all the test runs in Katalon Studio will be uploaded to TestOps automatically. However, the ones BEFORE the enabled integration are NOT.

That means this test suite result - I ran before the enabled integration - hasn't been uploaded to TestOps yet. To do so, manually select Katalon TestOps > Upload > Choose Team and Project > Then Upload. Wait for a few seconds... Refresh to fetch the change... Ah, our test result has been successfully uploaded - ID 255. Let's click on the ID to navigate and view the execution details directly on TestOps.

Here's the summary report for our test run #255... Since our tests passed, there's nothing to see in the Failed Test Results. Scrolling down to the Test Suite section, click on it and you can see the historical executions of this particular suite. Click on Test Results, it will show you all the executions in this test run. You can search for particular tests or filter them by their status, test suites, time/duration, and profile.

For each test, it lists out all the executed steps as you're seeing here in the Logs section. You can switch to Text view or remain in the Tree view mode, just like when working with the Log Viewer in Katalon Studio... There's also a download button in case you want to save it in your PC or send it to others. Right above is the Failed Assertions where it would explain why the test failed. But since ours passed, ''there's no data to display.''

Alright, back to the Studio, let's run a failed test suite this time... And again, to save our time, skip it to the end of the execution. You can see here in the Job Progress, our test report is being uploaded to TestOps automatically. The same goes for every other test suite execution since the Studio-and-TestOps integration is enabled, as I've mentioned earlier. It's actually done now! Let's head to TestOps and see how things are going there.

So, in TestOps' Reports category, our report has been uploaded successfully - ID 256... 1 passed, and 1 failed test. In this Test Runs tab, you can see the summary details of the failed test result. Click on it and you can further investigate the root cause with Error Message, Screenshot Attachments, and Test Logs. Another cool thing about using TestOps' reporting is that you can link, report bugs on Jira, and even assign a maintainer - who will fix this - directly from here. We'll discuss these in detail for another course.

Now, let's move on exploring TestOps a little bit more... You can go to Dashboard and view all the recent test activities for a particular project - this week or the week before. The TestOps team is working on rolling out new features and enhancements soon, so stay tuned.

Continue! Looking at this week, we have 2 test runs, 2 test cases, so 4 test results. Based on the charts and colors, we can tell there were:

- 3 passed and 1 failed test results

- And equally 1 passed and 1 failed test runs

Clicking on View All on the bottom right corner, you'll go to the Reports category - where you can view all the test runs for this particular project. Let's quickly walk through this together, shall we?

There is a chart here summarizing the pass/fail trend of your most recent test runs, dated back to 28 days max. Scrolling down, you can see all historical test run data, up-to-date. The first one is the latest test run that we went through just a moment ago. And in case you're wondering what these letters stand for:

- P means Passed

- F - Failed

- E - Error

- And I for Incomplete

Moving on to Summary by Test Run Results... here you see the overall status and performance of your test cases. This chart gives you insights on the trends of your test executions over time, how many test results and test cases passed or failed, along with their time duration. And below is the summary of test runs weekly overtime. Through all of these data, you can know more about the performance and quality of the application under test as well as the automated tests themselves - like how how flaky they are and the execution trends.

Well done! That's the end of our course. To recap, here's what you've learned:

- How to work with Log Viewer and Reports in Katalon Studio

- Exporting and understanding reports in different formats like CVS, HTML, and PDF

- Setting up to send test report automatically via email

- Finally, utilizing TestOps' advanced reporting capabilities to centralize all test execution data and get more insights on your test performance over time

So, what's next? Practice makes perfect! Go to Download on Katalon.com, install Studio, and start applying what you've learned in this course into your project. Katalon Academy is working on even more courses to help you learn about test automation and immediately put them into practice with Katalon tools. In the meantime, you can watch more of our existing courses by clicking on the Courses button right on the top menu bar. Thank you, and happy learning!