Katalon TrueTest: Proof of Concept Process

Additional notes:

This will show you how to configure a TrueTest Agent for an application under test.

It is very important to set up the tracking domains correctly. You don't want to mix the data from different application versions and/or different environments.

If the customer wants to add multiple domains in the configuration, please ask why.

Anti-practices:

1. Collect data from multiple test environments at once (QE, Staging, Pre-prod): it is likely that each env has a different version of the app and different data set. If they use data from those app versions and env, the test cases will have both old and new app versions mixed up in the same set. It will take extra time for us to analyze the root cause when they fail.

2. Collect data from multiple production sites which serve different purposes. For example, an e-commerce customer during our beta program used the same code base for both their B2B and B2C sites with different URLs. They originally configured their TrueTest agent to receive data from both sites. It was extremely hard for us to identify which cases belonged to the B2C site and to the B2B site to provide correct data and execute in the right environment.

When can we add multiple domains to the same agent? If the customer testers are testing all of these domains as part of the same application. Examples: when the application consists of multiple smaller applications hosted on different domains and they are linked as part of bigger end-to-end flows.

Additional notes:

Some challenges you might experience:

1. The TrueTest agent is not activated even after you have recorded some activities. Some possible causes: incorrect client code used with the extension; did not start the extension when recording; there might be some delay between the data capturing process and the activation process.

2. You cannot generate a user journey map due to "no data found during the specified time frame". Possible causes: the time frame selected did not include your sessions due to time zone difference; there might be some delay between the data capturing process and data query process.

3. You recorded multiple scenarios but there were a smaller number of user sessions recorded. Possible causes: you might have forgotten to close the browser tab to end a user session.

Additional notes:

Some challenges you might experience:

1. There are less number of nodes than the number of pages you interacted. Possible cause: some pages include dynamic patterns in their URL segments; hence, they were grouped into one node. Example: https://cellphone.aut.katalon.com/product/apple-iphone-15-silicone-case-with-magsafe?category=99 and https://cellphone.aut.katalon.com/product/apple-20w-usb-c-power-adapter?category=123 will be grouped under "/product/*".

2. There are fewer identified flows than in your scenarios. Possible cause: the scope covered by all the identified flows also covers the scope of the missing scenarios.

Additional notes:

Some challenges you might experience:

WebUIElement Not Found error. Troubleshooting steps:

1. Check if the element still exists in the application on the test environment. If not, the test case failed correctly, as the application has changed since TrueTest captured the data.

2. If the element still exists in the application, check if its locators are correct and if Self-healing is enabled for the project. On rare occasions, the user will need to update the locators manually. Please report the issue to the TrueTest product team so that we can enhance the data tracking and locator generation process.

3. Check if the current user login has the necessary permission or data to view the element.

4. Check if the flow to the element is still valid, as the steps might be routed differently.

Additional notes:

Expected Outcomes of Technology Compatability Validation:

1. User journey maps and flows reflect what you did in your sessions.

2. The map is not useful for single-page applications as they only have one URL, one node. If the test objects and test cases are still good, the customer might still want to proceed with the POC.

3. Test objects are captured with reasonable names and locators.

4. Review test object names and locators to see if the AI can recognize those objects well. If you see weird names or locators, use the browser tool to inspect the corresponding UI elements to see if they are well-defined.

5. Check if you need a reasonable amount of effort to get the test cases to execute.