Chapter 1 - Understanding Data-Driven Testing
Chapter 2 - Data-Driven Testing With an External Data Source
This course demonstrates the process of setting up and applying Data-Driven Testing with Katalon Studio. By using just one test script to run with different data from multiple sources, you can increase the test coverage with fewer resources and maintenance costs.
In three chapters, you will learn about:
Chapter 1 - Understanding Data-Driven Testing
In this first chapter, let's quickly go over the concept of Data-Driven Testing and the benefits it offers for test automation. Then, we'll move on to using Katalon Studio to set up and run data-driven tests. Let's start off with this scenario. When applying automated testing to verify a login screen's functionality, you need to perform tests using dozens of accounts, even hundreds, with different sets of credentials and user roles to increase the confidence level when shipping this login screen to production.
If you're new to test automation, here are several approaches that you could think of. The first one, you create ONE automated test script and manually change the credential values every time you run the test with a different account. So, 100 accounts mean 100 test runs and 99 times of changing data manually before running the tests. This sounds like the worst solution ever!
The second approach, you create an automated test and then duplicate it. Each duplication is for an account that you're going to test with. So, 100 accounts equal 100 test cases... Put them into a test suite and run. This is still very time-consuming and results in a lot of duplicate scripts. Not to mention that when the UI changes, you need to update all of the test cases... This approach isn't better than the first one.
The third approach, and also the right one, is to apply Data-Driven Testing. Meaning you create ONE automated test case, using variables for inputs and expected results, and prepare an Excel file to contain all the test data. If you want to test with 100 accounts, then the file will have 100 rows of credentials. What you need now is to code the script to get the test data from the Excel file and run the tests one by one. And that's the concept of Data-Driven Testing - running the same automated test case or test script with different test data that can be stored and read from a separate source, like an Excel file or database.
Then, why Data-Driven Testing? Well, first of all, more test data means higher test coverage and higher chances of bug detection. Second, Data-Driven Testing saves up time and resources since you don't need to create and run many test cases for each set of test data, hence, minimizing the duplicate codes and work. Also, it's easy for you to maintain. Test data, including test inputs and expected outputs, are stored separately from test scripts. You can change, update, or add new data to the data file without changing anything in the script.
Moving on to how Data-Driven Testing really works. Based on the concept I've mentioned earlier, you might have guessed it. Instead of using hard-coded values, a test script reads and uses test data stored in external files or database management systems to test a given AUT.
For Data-Driven Testing, Katalon Studio supports various types of data files to which you can bind variables, including internal data in Katalon Studio, external data files such as MS Excel and CSV files, and any Java Database Connectivity (also known as JDBC) sources like PostgreSQL, MySQL, Oracle, and MS SQL Server. That's all you need to know about the concept. In the next lessons, we're going to show you how to create an automated test case and bind data to apply Data-Driven Testing using Katalon Studio. See you.
Hello. Here I have my Katalon Studio ready. Just a quick reminder, you can go to Katalon.com/download, install the tool, and follow along. Let's start by creating a new test case. Right-click on the Test Case folder > New > Test case. Give it a name. For this course's demos, we're going to test the search functionality of the Airbnb website. So, I name mine ''Find a place.''
With Katalon Studio, you have several ways for you to automate your tests depending on your programming skills and experience. You can use Record & Playback, built-in keywords in the Manual mode, or using the Script mode to write your test script.
In this case, I'll choose Record & Playback. Click on the Record Web button on the toolbar. Paste in the URL to the Airbnb website. Then, hit Record. It will open a browser window and navigate to the Airbnb website. Every manual action you take and every object you click on will be recorded and turned into automated test scripts. Let's search for Atlanta. Check-In... 3 days. Then, select Search... Now, I'll choose this text ''stays in Atlanta'' for verification. Now, stop recording.
Here we have all the recorded steps. Everything seems good. Let's go ahead and Save Scripts. This window pops up, giving you the options to save these new test objects. On the right is the folder structure of my existing repository. On the left, you can see the test objects are organized under a Page-Object model. I will create a new folder to save them. Name it ''Airbnb pages.'' Select it and click on OK.
You can further rename the pages in the object repository for better visibility and maintenance. I will rename this one to ''Page_SearchResult'' and the other ''Page_Search.'' You can also quickly locate a test object by double-clicking on it (in the test case). For instance, this ''Check-in'' object is in the ''Page_Search'' folder.
Alright, I'll run this test to see how it goes. What I just showed you is basically how to use Record & Playback to quickly automate a simple test. To learn more about this low-code approach, you can check out our Katalon Studio: Record & Playback course. Well, after finishing this course. Our test run is finished. And it passed.
Now, what if you need to test with more test data - with more cities? You can double-click on the Input column in this test case and change the value. Make it ''Tokyo'' for the city input. Change it to ''Stays in Tokyo'' for verification. And, let's run this again.
This is the first approach we've mentioned earlier. You change the value in the test case every time you want to test with new data. But it's really time-consuming and seems to go against the core benefit of test automation - which is supposed to save time and resources.
Okay, so this is when you need to apply Data-Driven Testing. To do this, your test case needs to use Variables instead of hard-coded values. Select the test case. Click on the Variables tab. Select Add. This first variable is for city input. I will name it ''city'' and type in ''Atlanta'' for the default value. Add another one used for verifying. I name it ''expectedResult'' and ''Stays in Atlanta'' for the default value. We use this value because the input city for searching is Atlanta.
Back to our test case. Click on the Input ''Tokyo'' here, change the Value Type from String to Variables. Select city for the value. And do the same for the verification step. I will change it a bit, switch the value type to Variables, and choose expectedResult. Don't forget to save it. I'll run this test and see how it works.
It will use the default values added for the 2 variables to fill in the location and verify the text on the search result page... So, it passed, as expected. The first step is done, setting up variables. Let's move on to put this test case into a test suite and continue to prepare our data file. At the moment, the current Studio version I'm using, 8.2.5, only supports Data-Driven Testing at the test suite level. However, in the next versions, from 8.3.0 onwards, it will also support the test case level. That means you will be able to bind test data to both test suites and test cases, depending on your test scenario and needs.
Alright, I have already created a suite here. Click Add and choose the test case to put it in. Next, click on Show Data Binding to map test data with the corresponding variables. You can see that the two variables we created have already been loaded for the Variable Binding. Right above is the Test Data section. Since we haven't created anything yet, it's all blank.
To create a data file, right-click on the Data Files folder on the left > New > Test Data. Give your data a name. I name it ''100 cities''. Here we have many data types. But let's go for Internal Data first. Then, choose OK. It will open a tab for us to work on our data. Click on the Add icon to create a new data column. This first column is for the city input. I name it ''city". For the second column, I name it ''result'' - which will be used for verification. After that, add in some values for these columns. Let's go for 3 cities: ''Atlanta''... ''Stays in Atlanta'', ''Tokyo''... ''Stays in Tokyo'', and ''New York''... ''Stays in New York''. Again, don't forget to save it.
Back to our suite and data binding. In the Test Data section, click Add. Choose 100cities data, which we've just created. Click OK. As our data has been added, continue to select Map all. It will bind the variables of the test case to the data column with the exact same name in the data file. Here it says, ''Found 1 match(es).''
Looking at the first city variable, it has been bound to the data in the city column. But the expectedResult variable is still the same. This is because it cannot find the column in the data file that has the exact same name as the variable. Our column name is ''result,'' not ''expectedResult.''
If it's just 1 or 2 variables, you can choose to do the mapping manually. First, choose the Type. Here we have several options. Let's go over them quickly. If you leave it as Default - it will use the variable default value in the test case to run data-driven tests. Data column - it will take the column name as value. Data column index - it will take the column number as value. For example, in the last value column, if I type in 2, it will bind the variable to the second column in the data file. For script variable, we'll skip it for now as it involves a Variable Value Builder.
I'll go for Data column. Next, select our '100cities' data and finally, choose the result column for the value. Save it and run this test suite to see how it goes. This test case will be executed 3 times, each time for a data set, or we can say there will be 3 iterations. An iteration is a test case executed with a row of test data. The first test is looking for a stay in Atlanta... Done and close. Then, it opens a browser again and continues with Tokyo. Lastly, New York. All the steps are mostly the same.
The test run is done and now it's preparing the report and uploading it to Katalon TestOps. Looking at this, all of the tests passed. You can see more details below under the Log Viewer tab. Run 3/3 - Passes: 3 - Failures: 0. You can click on the Result tab to see the executed tests' report. Since I've already set up video recordings, here I also have an option to rewatch all the executions. Or you can click on the Report folder on the left, select the executed suite to see the same report.
We have a separate course for using Katalon Studio for test reporting and TestOps integration. Make sure to check it out if you want to know more about working with test logs and other features for troubleshooting. Alright, that's the end of our chapter. You have learned how to use variables for your tests, create internal data, and map them together to apply Data-driven testing. In the next lessons, let's try out Excel and CSV files and also JDBC databases. I will also show you how to combine data from these sources to run data-driven tests. See you.
Chapter 2 - Data-Driven Testing With an External Data Source
Hi, welcome back! In the last chapter, you've learned to run data-driven tests with internal data in Katalon Studio. However, in many cases, you'll need to use different data files from other teams to run your tests, such as developers, clients, marketing, sales teams, and so on. So, having the option to import and use external data sources would be convenient and save up lots of time for data setup. In this video, I will show you exactly how to do so - using external data files to run data-driven tests with Katalon Studio.
The steps are quite the same as in the previous lesson. First, create a new test suite - ''DDT with MS Excel Files''. Then, add a test case. I will use the same one which has already used variables. Next, we need to prepare our test data. Open your Excel file on your computer. Fill in the data, just like when we were creating the internal data in Katalon Studio. You should give the data columns the same names as the variables in your test cases so that you can map them easily. Alright, let me fill in the data real quick... First, ''Atlanta''... Second, ''New York''... And lastly, ''Tokyo''... Save this file.
Then, go back to Katalon Studio, create data in the Data Files folder. Give it a name. Here I name mine ''excelfile_100cities''. A tip here: you should get a consistent format for naming your data files to find and manage them more easily. Alright, keep the type as Excel because that's what we want. Click OK. Continue to click Browse and select the Excel file . You can also choose the sheet you want to use. Save it and move on to data binding.
Select the test case. Click Add in the Test Data section and choose our Excel data file. Then, click Map all. The two variables have been bound to the two data columns that have the same names. Save it and run this test suite. It's going to run the tests according to the test data order in the Excel file. First, it searches for Atlanta... Now, it goes for New York... And lastly, Tokyo. All of our tests passed. Easy, right?
The good thing about Data-Driven Testing is you can add more test data without changing anything in the test case or test script since it is stored separately. Let's say, I want to test with another city. All I need to go to the Excel data file and add it in. For example, here I type in ''Miami'' for the city input and ''Stays in Miami'' for verification. Save this file. In case you run your test cases with CI tools and store your automation projects in a version control system like Gitlab, you will need to commit the data file change you made. Otherwise, it will use the old file to run the tests.
Let's run the test suite again to see whether it will automatically update the new test data. What we're expecting is after running the tests with Atlanta, New York, and Tokyo cities, it will continue with Miami, which we've just added. Let's wait for a bit... Ah, here it is. It runs for the 4th time and searches for Miami. All done. The report shows there are 4 ''test cases'' which are actually 4 iterations. The same goes for the Log Viewer below: Atlanta, New York, Tokyo, And Miami. That's how you apply Data-Driven Testing with Katalon Studio using an Excel file.
Another type of file you'll likely encounter for storing and retrieving data is CSV. It is a simple text file containing records separated by a delimiter, often a comma. Due to the simplicity and widespread use across different platforms and systems, CSV files are a preferred format for data-driven testing.
With Katalon Studio, the steps are similar to how we use an Excel file. First, create or get a CSV file. Then, create a data file in Katalon Studio and import it in. Remember to choose the CSV type and select the separator the same as the file. Next, bind variables to the test data. Finally, run the suite. That's how you apply Data-Driven Testing with an Excel or CSV file. In the next lesson, we're going to show you how to run data-driven tests with an external database and how to bind variables to different data sources. See you.
Hi! In this lesson, I'll show you how to configure and run data-driven tests with an external database. Currently, for Data-Driven Testing, Katalon Studio supports many databases which use Java Database Connectivity (JDBC for short). If you want to use any other one, just need a JDBC driver. You can check the latest updates in Katalon Docs to get the compatible database management systems and Katalon Studio versions.
For this demo, I will use Postgres, also known as PostgreSQL. It is a free and open-source relational database management system that emphasizes extensibility and SQL compliance. To set it up, you can go to postgresapp.com, follow the instructions and learn how to install a Postgre DBMS from there. You can also download and install pgAmin from pgadmin.org/download to manage your database more easily. You should skip this precondition step if you already have it.
Here, I've already installed and set up my local Postgres and pgAdmin to manage it. Now, I'm going to prepare the data table. Go to Schema > Choose your schema, mine is Public > Select Table. Give it a name. Then, click on Column > Add row. I'll fill in some names here. Click Save. Next, I'll add values to these columns. Right-click on the table > View/Edit Data > All row. Double click on the blanks and fill in the values. Like the previous examples, I will add 3 cities here... Atlanta... Miami... and New York. Then, click Save on the toolbar.
To check, I'll write a query to get the data from this database. In the Query Editor, I type in the statement ''Select city, expected from db_ddt'' then hit the Run button. Here it is, our data table. Everything seems good. Let's head back to Katalon Studio.
Go to the Data Files folder to create test data. I name it ''DB_100cities'', following the name format like the previous ones. Select Database data for the Data Type. Then, click OK. After the data file is created and opened, select Edit Query. In this Database connection and Query Settings window, we need to configure to connect Katalon Studio with our Postgres database and write the query to get the test data set.
First, fill in ''PostgreSQL'' for the JDBC driver. Copy the URL sample below for our PostgreSQL and paste it in the Connection URL section. Next, we need to replace the ''DB_NAME'' and ''SCHEMA'' components in the sample with the values in our database. Switch to the pgAdmin, here are our database name - ''postgre'' and the schema - ''public''. But let's change that to ''katalon_studio''. Then, type them in our URL sample... Press Test Connection, and it succeeds, ''Connection successfully.''
The next thing is to write the query, just like what I did earlier, ''select city, expected from db_ddt''. It will find the data according to the query and load it in. Let's save it. From here, the steps are similar to the previous lessons.
Create a test suite and add a test case. Then, move on to data binding. Add in our DB_100cities data file and press Map all. The first variable has been bound to the city column in the database (which has the same name). Meanwhile, the second variable has not since the column's name is different. I'll do this mapping manually by selecting Data column for Type, DB_100cities for Test Data. And expected for Value. Save it again and run this test suite.
According to the data order, it will run 3 times, with Atlanta, Miami, and New York. And just like the other data sources, when you need to add or change the test data, you just need to do so in the database. You don't need to change anything in the test case or test script. Let's wait for a few seconds until it's finished.
Alright, looking at the Job Progress, everything is almost done. It's preparing the report and uploading it to TestOps... Our test passed. All the execution details have been noted down here in the Log Viewer. You can download the report to your local computer and send it to others by clicking on the Result tab and selecting Export Report.
For example, I will download the HTML report for this test suite execution. You can see our test suite name here and summary information. There are 3 tests here - which are actually just 1 test case that ran 3 times with 3 different sets of data. In each test case, it displays the summary, status, executed steps, and data binding which gives you information on which test data is used. To learn more about these kinds of reports, make sure to check out our course: Katalon Studio: How to work with execution logs and test reports.
That's the end of our chapter. You've learned to apply Data-Driven Testing with different external data sources, including Excel, CSV, and now databases. Moving on to the next lesson, you will know how to combine data from different sources, check data change before running your data-driven tests, and why you should do these. See you.
Chapter 3 - Data-Driven Testing With Multiple Data Sources Combined
Well done. You've reached the last chapter of this course where we're going to learn how to combine data from multiple sources to run data-driven tests. This is useful for large and complex projects where you need to gather many data files from different teams to run your tests.
Alright, let's create a new test suite and then add a test case that has variables. In the Test Data section, click Add to select the data source. But instead of choosing only one, I will go for both Excel and CSV data files. Click OK. Simple. But let's go for three. I'll also add the Postgre database we've created earlier. So, we have 3 data sources here.
For Variable Binding, you can use these Set Type and Set Test Data buttons here to do the mapping. For the city variable, click on Set Type and choose Data Column. Continue to select Set Test Data and choose the Excel file. For the value, pick the city column. Do the same for the other variable. Choose Data Column. For Test Data, let's go with the CSV file. And for the value, select the expectedResult column. That's it for the setup. I'll run this test suite and see how it works.
It is expected to use the Excel file for the city location input and the CSV file for verification. Particularly, in this Excel data file, it will use the values in the city column: Atlanta, New York, Miami, and Tokyo respectively to input the cities for location. And in the CSV file, it will use this expectedResult column: Stays in Atlanta, New York, Miami, and Tokyo respectively to verify the search output. Since the city order for the input and output from the two data files match each other, I expect our test run will pass. Let's wait and see.
And it's done... All of the tests passed, as expected. Click on this Result tab if you want to see or download the report for this test suite execution. Now, I'll bind the expectedResult variable to the Postgre database. Run this suite again and expect it to fail.
Let me explain. The second and third values in this Excel file's column are New York and Miami. Meanwhile, they are Stays in Miami and Stays in New York in the database's column. Since the city order does not match, our test run will fail. Let's wait and see.
So, the second test has failed. Just like we expected. It failed at step 7, the verifying step. It could not find the ''Stays in Miami'' text because the input was New York, hence the search output was actually ''Stay in New York'' on the Airbnb website. That concludes our chapter. You've learned how to combine test data from different sources to run data-driven tests. In the last lesson, we're going to see how you can use the Checkpoint feature to check data change before running tests and why you should do so.
One of the benefits of Data-Driven testing is that test data is separated from test scripts. However, in large and complex projects, the data files you use might be modified or get updated by the teams in charge. This might reduce the data integrity and cause your tests to fail at certain data-bound steps. To detect this issue and save time on test executions, you can make use of the Checkpoint feature in Katalon Studio. You'll use it to verify whether the test data is still the same as the data source before using the data to run the associated steps.
To do so, select a test case > Add a step. Type in ''Verify Checkpoint'' and use this keyword. Next, you'll need to create a checkpoint. Go to Checkpoints > New > Checkpoint. Give it a name. I'll leave it as it is. Next, choose the type. I'll go with Database Data. Click Next. Here, we'll need to configure the database connection, just like what we did. When everything is done, select Finish.
It will take the data from the database according to the query I wrote. Select the values you want to check. I'll choose all of them. This means if one piece of the selected data is changed compared to the database, the tests that use these data will fail at the Verify Checkpoint step. In real-life scenarios, you shouldn't check on all test data. Instead, only check on the key and important data to minimize the chances of test failures due to data change issues.
Next, double click on the Input of the Verify Checkpoint step. Change the value here from null to the Checkpoint we've just created. Select OK. Now, if I run this suite, nothing's gonna happen since there's no change in the database. So, I'll go on and modify it a little bit to help you see the outcome of using Checkpoints. After navigating to the data table, I'll delete the ''s'' in the ''Stays in Tokyo'' text... Don't forget to save it.
Here's a tip I usually do to save up execution time and resources whenever I need to check data change for Data-Driven Testing. If all the test steps are good and stable (you can tell from previous test runs), then put the Verify Checkpoint step at the beginning of your tests. To see what I mean, let's save and run this test suite.
Before opening a browser, it will go and check on the selected data. If there's any difference compared to the database, the verifying steps will fail, causing the tests to fail. Let's wait for a few seconds. And it's done. You can see there are 4 test failures. They all failed and stopped at the 1st step verifyCheckpoints due to the data change - ''Stay in Tokyo.'' You don't have to wait for several steps to be executed just to find out your tests failed due to this issue. This means you detect it earlier and save up execution resources.
In case you wonder why there are 8 test runs, instead of 4 like usual, it's because of my Retry settings for test failures. You don't need to care about that right now. We'll mention it in another course. Alright, congrats! You've completed this course. You've gone through everything to start applying Data-Driven Testing with Katalon Studio.
To recap, you've learned the what, why, and how of Data-Driven Testing, how to apply Data-Driven Testing using Katalon Studio with internal and external data from Excel, CSV files, and databases, combining data from multiple sources to run data-driven tests, and finally using Checkpoints to check data change before running tests.
So, what's next? Practice makes perfect! Go to Download on Katalon [dot] com, install Studio, and start applying DDT to your projects. Katalon Academy is working on even more courses to help you learn about test automation and immediately put them into practice with Katalon tools. In the meantime, you can watch more of our existing courses by clicking on the Courses button right on the top menu bar. Thank you, and happy learning!
NEXT COURSES TO LEARN