Refine your CI/CD Pipeline with Automated Testing

Into the CI/CD Pipeline: What It Is, Why It Matters

00:05-00:41

So I'd like to start with one key code that I like to use over and over. I'm personally a fan of motor racing and as I'm sure most of you are, a lot of you are. But one of the things that stuck with me that Mario Andretti, who's a former NASCAR race car driver for those of you who don't know, he had a very interesting quote a few years ago where he said, ''if things seem under control, you're just not going fast enough.'' Right, I immediately identify and associate with that just as a tester.

00:42-01:47

In any of the engagements, any of the projects that I used to work in and I've continued to work in, I've always had that challenge given to me either by my development lead or the project manager or the scrum master, saying how can you actually do to speed up the testing process. And I think with the availability of CICD tools for us, I think this is something that is becoming a reality now where as things seem to pick up the pace and we are able to test more efficiently in terms of speed. You know it's always pushing the development to the next level right to say ''hey how can we even get faster than what we are doing now?'' and you know that's a great topic to continue talking about. What besides CICD can we as testers do to actually increase the speed?

01:48-02:21

Coming back to CICD right. One of the key things that we have to always think about is why do we need test automation in our solution right? Why do we need to apply and leverage CICD tools? So if you take a step back and you think about really what is happening within the industry right now, I think one of the key observation you will see is the global competition with software providers, right?

02:22-02:53

The global competition has basically allowed the end-users of those systems, those applications, to have a very low tolerance level for defects right? You know I can use an example of mobile apps. Studies have shown where people download mobile apps, if they don't like what they see within the first 45 to 50 seconds, they close the app, they uninstall the app and they're not coming back.

02:54-03:19

So you know that that is one example of low tolerance that has occurred simply due to just the availability of so many alternatives out there today right? And that's fueled really by the global sense of competition to get to market and get open as early as possible and that leads into the second aspect along with low tolerance for defects.

03:20-04:01

The other pressure that I think delivery teams and agile teams feel is how soon can we get a new iteration, new idea, a new feature, a better feature out in the hands of the users as soon as possible. So that has resulted in, you know, putting extra pressure on the agile teams to deliver in shorter and faster iterations as part of their product life cycle. From that perspective, you know as engineers, when we find and try to find the solution in the industry buzzword if you will, it has been DevOps.

04:02-04:37

I agree DevOps being as an integral part of this solution. But for teams that are not mature yet and have not gotten a lot of the DevOps toolchains in place, I would recommend starting uh with step one which is CICD - continuous integration continuous delivery. So really from a tester's perspective, what does that mean right? What this basically means is we have to test earlier and test more often.

04:38-05:11

Obviously, that can be accomplished very effectively by applying automation tools such as Katalon. The second aspect though is a little bit more tricky but what that is is basically how do we test earlier and test often, but how do we also continue to test real-world workflows, and what I mean by that is really the scenarios or the processes and workflows that are going to be really used by the end-user or audience of the system under test.

05:12-05:46

In order for you to do that it cannot completely be dependent just on your automation scripts or your automation tests, there has to be an aspect of manual exploratory testing that has to occur. However, if you do not leverage automation tests and leverage it in your CICD pipeline, you're in essence, as a tester, not gonna have a lot of time to actually go and do exploratory testing to add more value to the product being delivered out there.

05:47-06:11

So one key thing that I do see continuing to be missed as people adopt CICD and do real-world workflows, and testing in that regard, is testing with a comprehensive mindset. And what I mean by that is, you know, not just focusing on functional tests, and whether a particular screen looks good, whether an API service provides the appropriate information.

06:12-06:43

But also, look at things around performance, around usability, around accessibility. Things of that nature are also what make the overall user experience and their adoption level of the product, of your software, that much higher. So it's very important that as testers when we begin testing applications, we're not just limiting it to just the functional aspects.

06:44-07:14

Functionality is a core key component. But from an adoption perspective for the intended audience or target users of the system, we have to look at it from other angles as well. So let's look at a traditional or a typical CICD pipeline, what I have on the screen here is an example of where you may be able to apply different tools as part of your CICD pipeline when it comes to testing.

07:15-07:52

So as I'm sure a lot of you may be familiar already, if you think about you know a CICD pipeline, there are two components to it. The integration piece is essential as the code is being developed by different developers in the team. They will begin you know checking in their code in your code repository such as GitHub or what you have. And as all of that code is committed and merged back, the overall build process is kicked off as part of the CI pipeline.

07:53-08:30

Tools such as CircleCI, such as Jenkins, definitely build out the packages and then help validate the integrity of the code. We, as testers, can definitely use tools such as Katalon to jump in and supplement the testing that is going on right after the unit test right. So with our experience, we have used Katalon as part of the integration test as well as also performing a series of high-priority tests. Functionality validation and things of that nature are also within the CI pipeline.

08:31-09:00

Because the one key important aspect of that that you get with applying automation to your CICD pipeline is to be able to test sooner and helps give faster feedback earlier feedback to the developers if you encounter issues. So using that as a business priority, to integrate tests sooner is a key advantage that you can now benefit from and apply using Katalon.

09:01-09:37

The second half of this is the continuous delivery pipeline, right so that's essentially where once the code is packaged and built and is ready to be deployed into different environments. You actually could use a tool such as a docker and Kubernetes and Puppet to orchestrate and have all of that created for you. And the CD pipeline is in essence another area where you can then reapply a different suite of tests from Katalon and make sure that your validation and things of that nature are effectively done as well.

09:38-10:19

The good part of this sort of layout is, you know, even though you have dedicated environments that you call out as QA or staging or production in essence by using an approach such as applying Katalon into like containers and such, you are stepping away from having a finite set of these environments. You can actually have categories of environments where you can group them as pre-prod or prod. And the number of environments under each of these two categories is totally related by how many category environments you can actually support your infrastructure.

10:20-10:45

So here is an example of how I would recommend setting up automation in your CICD pipeline. Right, one of the key important things for you to think about is how do you logically organize and group your test cases and test suites. Do you tag them, do you associate them with a certain code branch, or tag them in certain functional modules of your code?

10:46-11:29

Just so that way you can logically organize and be able to pull up a specific suite of tests depending on the area of the system, or the core that is being modified. So having a logical group is important. The second aspect of that because of the way you actually can now logically group these test suites, you can apply rules within your build server and also on your repositories to be able to dynamically select and choose what test suites are required to be executed.

11:30-11:58

Katalon also has the capability of tagging and then also creating dynamic test suites based on certain triggers so you can definitely leverage that aspect as well. And then once all of that is identified, you can configure that with the CICD server to be able to automatically execute a bigger suite or a smaller suite or a specific suite based on the type of code changes and commit that have been done.

11:59-12:24

And what that actually allows you to do is mitigate a bit of risk to say you know what is the impact of the direct changes to the code that have occurred so far. And then you also get the flexibility of creating a parallel effort of a bigger regression suite if that's what is needed. And once you have all of that configured, you can trigger and run the test based on those conditions.

12:35-12:50

The great part of this is because it's part of an auto kickoff and auto selections, the key analysis aspect comes in where you may, at this point, needs some manual oversight people analyze it. The great part about where our industry is headed is that could technically also potentially, in the very near term, be supported by AI and ML algorithms as well.