Chapter 2 - Benefits of Using CI/CD Pipeline
Chapter 3 - How to Run Katalon Tests on Jenkins
Chapter 4 - How to Run Katalon Tests on CircleCI
Many software companies have rigorously applied Continuous Integration and Continuous Delivery (CI/CD) to release higher quality products faster and quickly respond to customer demands. To generate faster feedback loops, automation testing, particularly a decent automation strategy, would surely be needed.
In this course, we’ll walk you through this procedure and how to leverage Katalon Studio’s automated testing features for an effective CI/CD implementation.
In these four chapters, you will learn:
Chapter 1 - Introduction CI/CD Pipelines and An Automation Testing Setup
So I'd like to start with one key code that I like to use over and over. I'm personally a fan of motor racing and as I'm sure most of you are, a lot of you are. But one of the things that stuck with me that Mario Andretti, who's a former NASCAR race car driver for those of you who don't know, he had a very interesting quote a few years ago where he said, ''if things seem under control, you're just not going fast enough.'' Right, I immediately identify and associate with that just as a tester.
In any of the engagements, any of the projects that I used to work in and I've continued to work in, I've always had that challenge given to me either by my development lead or the project manager or the scrum master, saying how can you actually do to speed up the testing process. And I think with the availability of CICD tools for us, I think this is something that is becoming a reality now where as things seem to pick up the pace and we are able to test more efficiently in terms of speed. You know it's always pushing the development to the next level right to say ''hey how can we even get faster than what we are doing now?'' and you know that's a great topic to continue talking about. What besides CICD can we as testers do to actually increase the speed?
Coming back to CICD right. One of the key things that we have to always think about is why do we need test automation in our solution right? Why do we need to apply and leverage CICD tools? So if you take a step back and you think about really what is happening within the industry right now, I think one of the key observation you will see is the global competition with software providers, right?
The global competition has basically allowed the end-users of those systems, those applications, to have a very low tolerance level for defects right? You know I can use an example of mobile apps. Studies have shown where people download mobile apps, if they don't like what they see within the first 45 to 50 seconds, they close the app, they uninstall the app and they're not coming back.
So you know that that is one example of low tolerance that has occurred simply due to just the availability of so many alternatives out there today right? And that's fueled really by the global sense of competition to get to market and get open as early as possible and that leads into the second aspect along with low tolerance for defects.
The other pressure that I think delivery teams and agile teams feel is how soon can we get a new iteration, new idea, a new feature, a better feature out in the hands of the users as soon as possible. So that has resulted in, you know, putting extra pressure on the agile teams to deliver in shorter and faster iterations as part of their product life cycle. From that perspective, you know as engineers, when we find and try to find the solution in the industry buzzword if you will, it has been DevOps.
I agree DevOps being as an integral part of this solution. But for teams that are not mature yet and have not gotten a lot of the DevOps toolchains in place, I would recommend starting uh with step one which is CICD - continuous integration continuous delivery. So really from a tester's perspective, what does that mean right? What this basically means is we have to test earlier and test more often.
Obviously, that can be accomplished very effectively by applying automation tools such as Katalon. The second aspect though is a little bit more tricky but what that is is basically how do we test earlier and test often, but how do we also continue to test real-world workflows, and what I mean by that is really the scenarios or the processes and workflows that are going to be really used by the end-user or audience of the system under test.
In order for you to do that it cannot completely be dependent just on your automation scripts or your automation tests, there has to be an aspect of manual exploratory testing that has to occur. However, if you do not leverage automation tests and leverage it in your CICD pipeline, you're in essence, as a tester, not gonna have a lot of time to actually go and do exploratory testing to add more value to the product being delivered out there.
So one key thing that I do see continuing to be missed as people adopt CICD and do real-world workflows, and testing in that regard, is testing with a comprehensive mindset. And what I mean by that is, you know, not just focusing on functional tests, and whether a particular screen looks good, whether an API service provides the appropriate information.
But also, look at things around performance, around usability, around accessibility. Things of that nature are also what make the overall user experience and their adoption level of the product, of your software, that much higher. So it's very important that as testers when we begin testing applications, we're not just limiting it to just the functional aspects.
Functionality is a core key component. But from an adoption perspective for the intended audience or target users of the system, we have to look at it from other angles as well. So let's look at a traditional or a typical CICD pipeline, what I have on the screen here is an example of where you may be able to apply different tools as part of your CICD pipeline when it comes to testing.
So as I'm sure a lot of you may be familiar already, if you think about you know a CICD pipeline, there are two components to it. The integration piece is essential as the code is being developed by different developers in the team. They will begin you know checking in their code in your code repository such as GitHub or what you have. And as all of that code is committed and merged back, the overall build process is kicked off as part of the CI pipeline.
Tools such as CircleCI, such as Jenkins, definitely build out the packages and then help validate the integrity of the code. We, as testers, can definitely use tools such as Katalon to jump in and supplement the testing that is going on right after the unit test right. So with our experience, we have used Katalon as part of the integration test as well as also performing a series of high-priority tests. Functionality validation and things of that nature are also within the CI pipeline.
Because the one key important aspect of that that you get with applying automation to your CICD pipeline is to be able to test sooner and helps give faster feedback earlier feedback to the developers if you encounter issues. So using that as a business priority, to integrate tests sooner is a key advantage that you can now benefit from and apply using Katalon.
The second half of this is the continuous delivery pipeline, right so that's essentially where once the code is packaged and built and is ready to be deployed into different environments. You actually could use a tool such as a docker and Kubernetes and Puppet to orchestrate and have all of that created for you. And the CD pipeline is in essence another area where you can then reapply a different suite of tests from Katalon and make sure that your validation and things of that nature are effectively done as well.
The good part of this sort of layout is, you know, even though you have dedicated environments that you call out as QA or staging or production in essence by using an approach such as applying Katalon into like containers and such, you are stepping away from having a finite set of these environments. You can actually have categories of environments where you can group them as pre-prod or prod. And the number of environments under each of these two categories is totally related by how many category environments you can actually support your infrastructure.
So here is an example of how I would recommend setting up automation in your CICD pipeline. Right, one of the key important things for you to think about is how do you logically organize and group your test cases and test suites. Do you tag them, do you associate them with a certain code branch, or tag them in certain functional modules of your code?
Just so that way you can logically organize and be able to pull up a specific suite of tests depending on the area of the system, or the core that is being modified. So having a logical group is important. The second aspect of that because of the way you actually can now logically group these test suites, you can apply rules within your build server and also on your repositories to be able to dynamically select and choose what test suites are required to be executed.
Katalon also has the capability of tagging and then also creating dynamic test suites based on certain triggers so you can definitely leverage that aspect as well. And then once all of that is identified, you can configure that with the CICD server to be able to automatically execute a bigger suite or a smaller suite or a specific suite based on the type of code changes and commit that have been done.
And what that actually allows you to do is mitigate a bit of risk to say you know what is the impact of the direct changes to the code that have occurred so far. And then you also get the flexibility of creating a parallel effort of a bigger regression suite if that's what is needed. And once you have all of that configured, you can trigger and run the test based on those conditions.
The great part of this is because it's part of an auto kickoff and auto selections, the key analysis aspect comes in where you may, at this point, needs some manual oversight people analyze it. The great part about where our industry is headed is that could technically also potentially, in the very near term, be supported by AI and ML algorithms as well.
Chapter 2 - Benefits of Using CI/CD Pipeline
So looking at what the three main benefits of having your automation tests as part of your CICD pipelines, the first one as I was mentioning earlier is all about getting the feedback of your success or failures or risks sooner or faster back to the development team.
Applying automation, in this fashion, also lets you broaden your test coverage by tackling different types of tests so you could do UI tests. You could do UI tests using headless browsers, you could do tests in the API levels, you could do validations of data and transaction information, and so forth. And ultimately, what that does is raise the overall confidence in the delivered software from your team. And as I was mentioning earlier, that is the key essence.
If you really think about it, the role of testers is is what is supplemented by this. Our role is to really increase the confidence in the software that has been delivered by the team. I also think the key benefit of getting automation here is really opening up capacity and bandwidth for testers to really go do the manual exploratory testing as well, getting a look at understanding the domain, and really applying that knowledge and that experience to really find issues that perhaps the core automation may not always find.
So key takeaways from this particular introduction, I would say are, you know, obviously, just manual testing is not enough to be able to respond to the demand and challenge of executing tests faster. The other aspect I would also shared with everyone is just automation through the UI is not sufficient. You definitely have to look at different ways of how testing and validation can occur and are not just limited to the UI aspect only.
And ultimately, you know, testing in CICD allows for both of these things to happen. And ultimately, the goal is freeing up your bandwidth of your tester for them to go ahead and do more exploration and find interesting issues that can only be done by really engaging with the application under test, and applying the domain knowledge that you build up.
Chapter 3 - How to Run Katalon Tests on Jenkins
Thanks so much for such a wonderful double part. You really explained what is the main importance of CICD in automation. So guys, I will continue with how we can implement the actual CICD from using Katalon. So in order to start with CICD, we have a tool called Jenkins, which is a CI tool. So, Katalon has released a plugin through which you can easily run your Katalon test via Jenkins. So this plugin is very useful when you have to integrate your test with a CICD pipeline.
So what exactly this plugin will do? So we have two demos for today. One is how this Katalon and Jenkins plugin works. And another demo is how CircleCI tool will fit for CICD. You can use any other CICD tool as well but we will continue with CircleCI. Okay, Katalon Studio is now capable enough to run different types of tests. So if you see right now, Katalon Studio supports mobile automation, API automation, web automation, right?
So, in this demo, we are running our web test. So we have a test which is simply testing few functionalities of an application and that test we will see how it will fit into our CICD pipeline. So we are using today Jenkins, Docker for deployment and we are also using today Katalon analytics to show you how it will update your analytics based on your test behavior, whether it's passed or failed.
Katalon is smart enough to integrate with all the tools like Jira, Slack, qTest, and many more. So what exactly we are going to do? Like how exactly Katalon will fit into your CICD? So as you can see, we have a build. So once your application is built. So let's say, we are building or deploying our application on the staging environment. Then your test will run. So, this test is nothing but the actual Katalon test and once this test is passed, then it will deploy to the production environment.
Let me go through it first. So we have a Jenkins job which we have created. What exactly this job does? So I'm assuming guys like you guys are familiar with Jenkins, which is actually a CI tool. So what we have is a repository. Just let me show you. This is our application which we are running. So I can also show you how exactly this application looks like once you deploy it and run it on the web.
This is our Github repository where actual our application is. You can see the application code and our test also is present. This is the actual repository. So here we have the application and the test also. So what exactly we are going to do so? If you see here right now, we have configured this job in such a way that if there is any change in the Github or any comment or changes done by the developers, automatically it should start your test.
So you can see in the build section, we have given a checkbox that shows 'Poll SCM'. If there is any change, it should immediately start your test. So what exactly this Jenkins job will do? First, it will deploy your application and once your application is deployed, then the second step, it will run your Katalon Studio test.
So we have a set of tests which are already predefined. So this is the command which will simply run your test. So if the application is deployed, it will run the tests. And how we are getting this option? It is coming from Katalon plugin which I've discussed just now. You just need to go to manage plugin, search for that plugin and you will be able to see this section.
So what you can see here right now, it automatically takes this version which you mentioned. And if you have any pre-installed version you can make mention here and it will take the pre-installed version as well. Right now, it is running on a Docker container. So it is running on a Linux system. So these are the commands which are coming and I will show you how to get these arguments. And these are some additional parameters which we are passing. Since we are running on a Linux system, we are setting up the resolution as well.
Now if you have been using the Katalon, then let's say, you have a test which is already configured and you have a test suite as well. Now let's say you want to run the same test suite. You can see this option on top. When you click on this option, it generates a command. So you can see here, this is the actual command which is giving you the completed detail about your test suite which you want to run.
So if you see closely, it says what is the run mode, test suite name, execution profile, browser type, everything. So the same command we have mentioned here. It is the only thing we have mentioned that we want to run the test in headless mode. Okay so this is what this job will do and if your job passes or fails immediately, it will reflect to the Katalon analytics and you can also see in email as well.
Okay so that we will see. So let me do one thing. Let me make some changes in the Github repository and this job should start. So let me make some changes. So this is the page or I will say the actual code where I have to make the changes. So we are going to change one small file. We are going to edit this. While you are working with any other editor, you can directly push the choice changes from your editor or from the CMD.
For this demo, I'm directly making the changes from the Github. So let's say, I made one change called login to make appointment test so it should immediately start your test. Just save this file, 'Yes'. And it will wait for few seconds and then it will start building this job. So you can see, it is running. So what exactly the job will do? As we have configured in this job, first, it will deploy the application and then it will run our Katalon test.
So, you can see our test also started. Since it is running on headless mode, you will not be able to see the UI and it is running on a separate Docker container. And if you just see closely enough, first part, it's all about deploying the application and once your application is started, then it will continue with your test part. And finally, if you see, our test passed. And this is just a few seconds ago, your test was also executed. And our test was expecting the same. So it was running as expected and it is passing.
Now let's do one thing. Let's make the change which is not expected and ideally this time, it should fail. So if you see here, we will make another change. So this is another file if you see. It's all about appointment application. When you book an appointment, you should get a confirmation that the appointment is confirmed. So, you can see there's one small text. So we are expecting the text should be 'Appointment Confirmation'. But let me change it to some other text and this time our test should fail because in our test, we are giving one assertion that it should verify this text.
Let me make this change. It will take few seconds and it will start building again. In the meanwhile, I can show you the email. So this was the test which we executed and it is working fine. The next test should start now which should fail. Okay, let's wait. And just a small thing which you want to see, it started by SCM. So as soon as we made the change, it is pulled by Jenkins and it is running.
Yeah, it's starting now. And if you see the logs, application is deployed. It is just checking the plugin folder. So actually, this test suite has two scenarios. Yes, our test failed. Okay, so you can see, it is saying that we are unable to verify the text which is 'Appointment Confirmation' because we did the changes. Now that the keyword is missing, it's not a complete keyword but yeah, that confirmation is not proper.
Now so our test is failing and it should fail. And immediately we'll get the result. One passed, one failed. And if you see the zip file which we got, just unzip this and see. So you can see, one test passed which is just a login test, and another test failed because of this keyword 'Appointment Confirmation'. And now if you make the change again, it will start and this time our job will pass.
So this is how you can simply configure your job. If there are any changes, it should start your test immediately. So let me close this part now. And immediately your analytics also should get updated. So this is fair now. Again, we made the changes now, it will run and this time, it will pass.
So the second part is now with a CircleCI, how you can deploy the application on staging, run your test, and based on your test status, whether it's past or fail, it should deploy to the production or not. Okay, so this is the first demo which we already discussed just now. What exactly is happening whenever a developer is making any changes, it is starting your Jenkins job and that Jenkins job is simply running your Katalon tests. Now let's just do another demo.
Chapter 4 - How to Run Katalon Tests on CircleCI
Yeah, so now second scenario is, let's say a developer is making any changes to the application, okay? So whenever a comment happens on the repository. So we have a CircleCI, which is a complete CICD tool, which will actually trigger a pipeline. Okay? So this pipeline will have the three stages. We'll talk about the stages now. First stage will be the build, which will build the application, and it will deploy on a staging system. Second stage will be running a Katalon test. So, this Katalon test will run your test. If your test passes, then finally it will deploy the same application to the production.
Okay, that is a happy scenario when everything is working. Your test is running and it will deploy to the production. Second use case, which we will see. That now developers making some changes and now these changes are causing some issues to the application, okay? Let's say some wrong commit happened or some additional code which is not tested. Then again, CircleCI will start the pipeline. Again, it will build application. Now, the Katalon test will start, okay? And if our test is failing, it will not deploy to the production environment.
That's what we are going to do now. We are going to see the actual configuration. What exactly we are trying to do here? Now, same thing you can do with any of the CICD tools. You can also do the same thing with Jenkins as well. But if you see, the look and feel of CircleCI is interactive, you will see the real-time execution. So this is actually YAML file of a CircleCI which actually has the three stages.
Okay, so if you see the first stage, what it does? It simply does the build part. Okay. It will build the application, so here, actually, by using a Docker Image. So, a lot of Docker Image will provide all the required infrastructure for that particular application. And finally, this command will deploy the application. Okay, so we have a Heroku platform. So this command will actually run or push our application and it will deploy. So, the first state is all about building, it will build the application.
Now, if you see the second stage, it is actually a test stage where we are again using Docker Image, which will simply install Katalon Studio on a runtime container. Okay. And finally, it will simply run the come out. So this is the same command which we have on the Jenkins, and which we got from Katalon studio as well. So there's the same command, which we are giving here. So what exactly it will do? It will simply run your test, and these are some additional parameters like it will store the result and artifact in a space, repository or particular path.
And the last stage is the deployment part. Again, we are using the same Docker Image for the deployment of the production, and it will do the deployment. So the minor changes if you see here, we are deploying on a staging environment. And here we are deploying on a production environment. So, this is what exactly our three stages. But this is the actual workflow, so workflow like you want to define how your build should, or the CICD pipeline should run, what will be the sequence.
So first it should build the application. Then it should go to the test and then deploy. Now, there we are giving one condition. The test requires a build. If the build is failing, it will not move to the test stage. In the same way, the deployment also needs test. If our test is sailing, it should not even move to deployment. So this is the dependency which we have given in the CircleCI. Okay, so this is actually you need to understand the syntax. That's all. Same thing you can do with others as well. Now, just let me quickly show you what changes we can do so that it should start. And let me also show you two applications.
So this is the actual staging application. Actually, it's the same application. We will make certain changes and it should reflect with the production. So, we have created two instances. One is staging. And the second one is the production one. Now, let's make changes. I will tell you some configuration, which is missing. So, let me first run this, and let's see. There's an authentication issue with Github, so maybe I have to log in with Bitbucket and try. Let's try it first.
I'm making some changes right now. Let's say, okay. There are some small minor changes right now. I've just logged in with Github. Let me log into Bitbucket, and try. So, I can tell you what exactly this build is doing. So that I can log in and show you, just a second. So this is actually three stages. What it is doing? It will build an application, test the application, and finally, it will deploy.
So right now it is at the build step because of some pull permission that we will fix it. So as you can see right now, the build itself is failings, it is not moving to test and deployment. So let me fix this. So guys, there are some issues with the building of the application. It has nothing to do with like Katalon test. Okay? So what we are going to do right now, I'm showing you are two builds which we executed like prior to the demo.
Okay, so I'm going to show you these two builds which actually work. Due to some recent changes in the application, now, the build is still is failing, okay? So now Katalon team is working on that, so it will be fixed in few minutes. In the meanwhile, I can show you what exactly the process is.
So this was a success built. So what exactly it does? Same thing, first, it will deploy application. Okay, so if you see the first part, it will set up your environment. Then finally, it will check out the code on that particular environment. And finally, it will deploy. And if you see the failed build, which we executed here. This one. So as soon as the build runs, okay, and if your test is failing, it will not deploy it to the production.
So now if we see why the test failed because we made some changes. Because of the changes, the test was failing and finally, it did not deploy to the production. Okay, so we made some changes in the Xpath, ID of that particular application. Okay. Due to that, it is not able to identify. So our test is failing and it did not deploy to the production. So that was the main intention of showing this pipeline, but some of the builds is still is failing right now. So it is not even moving to test and it is not even moving to deployment stage. Let me show you this one as well.
Yeah. Because the ones, if all the stages are working, then only your pipeline will be succeeded. So, build then test, and finally deployment part. And based on your build status, you will get keep on getting the mails immediately, and same, it will immediately be reflected to Katalon analytics as well.
So guys, excuse us for this particular build issue because the application was having some issues. So it is not even building but this pipeline is completely working fine based on our workflow. If the build is failing, it is not even moving to the next stages.