Since OpenAI came up with ChatGPT, rumors about the upcoming improvements of Artificial Intelligence are omnipresent. In this article, I want to discuss, if and how we can use chat-bots for software QA and Test Automation. Therefore I want to use ChatGPT. The tool is designed to give detailed answers to questions, using multiple Sources with the ability to summarize complex topics. Before I start to discuss how we can use this technology for Software QA, I want to point out how ChatGPT works. I am not an expert in the field so I had to ask someone who knows exactly what’s going on behind the scenes. I asked the only real expert within the field that I know and that is – you guessed it – ChatGPT itself.
For the Question of how ChatGPT works, I got a response that was a bit too long so I asked it to sum it up into two sentences. Here’s the response:
“ChatGPT is a transformer-based language generation model trained on a large corpus of text data to generate human-like responses to text inputs. It predicts the next word in a sequence by generating a probability distribution over the vocabulary, given the context of the previous words.”
The explicit focus on language is good news if you think of test automation, because the scripts that are used, are written in a programming language which is in fact a kind of language. Therefore I thought ChatGPT could be able to generate proper test-scripts. I decided to test this on my own. For a first short overview of ChatGPT’s skills in its understanding of human language, I gave it a clear instruction – at least it would have been clear for a human reading it. Here’s my little conversation with ChatGPT:
Me: “ChatGPT, please write a C# xUnit script that tests ChatGPT’s ability to calculate on a first-grade level. Test case: 1 + 2 = 3.“
ChatGPT: ”Here's an example of a simple xUnit test case in C# to test Chat-GPT's ability to perform first-grade level math:
using Xunit;
namespace ChatGPTTests
{
public class FirstGradeMathTests
{
[Fact]
public void OnePlusTwoEqualsThree()
{
int result = 1 + 2;
Assert.Equal(3, result);
}
}
}
Me: “This does not Test ChatGPT's calculation skills. If you have some trouble with mathematical problems I won't judge you. So please give me a proper test script.”
ChatGPT: “I apologize for the confusion earlier. Here's a corrected example that tests ChatGPT's ability to perform arithmetic operations, rather than relying on hardcoded values:…”
…followed by a script that tests different mathematical operations, but NOT ChatGPT at all…
Me: ”Ok next try... Please write a c# xUnit script, that navigates to ChatGPT, asks it to calculate 1+2 and checks if the result is 3.”
ChatGPT: “Here's an example of how you might write a C# xUnit test to interact with ChatGPT and check if it returns the correct result for the calculation:… “
Now the following Code goes in the right direction. It uses a placeholder for an API Endpoint which would give a response and ChatGPT explains that this placeholder has to be substituted by the “actual API endpoint”.
Let’s make a little recap at this point. I formulated my question in a fairly naive way to figure out if ChatGPT would be able to translate human language into a language that a machine would understand. This didn’t work out completely. Though ChatGPT was able to write a script that would compile, the script does not exactly do what it should do. The more precise my instructions got, the better the result was. This shows, that using a chatbot is a skill that you can improve. Maybe it’s a worthwhile idea, to train your ability of using these machines. Above all because of the expected progress of this technology.
Assuming that this progress will not only affect chatbots but the entire field of AI, other tools could participate in these advances. Chatbots are not designed for concerns of software QA. But other tools that are using AI are specially created to support the quality of the software development process. So let’s do a little research for which tools are those. But how can we do this kind of research? Well, this is exactly what ChatGPT is designed for, so let’s use it for this purpose:
Me: “ChatGPT, which Test-Automation Tools are using AI?”
ChatGPT: ” There are several test automation tools that incorporate artificial intelligence (AI) in various ways. Here are some examples:
These are just a few examples of test automation tools that incorporate AI. As AI continues to evolve and improve, it is likely that more test automation tools will incorporate AI in innovative ways.”
This is a very good result. If I was planning to implement AI-powered tools into my testing infrastructure, I now have got a short overview of existing solutions. As we have seen above, the more precise my requests, the more specific the answers are. If I was interested in a specific tool, I could start my research to get further information from ChatGPT.
Let’s try it with the code analysis tool CodeScene:
Me: ”What benefit does CodeScene have for my software development project?”
ChatGPT: “CodeScene is a tool that provides several benefits for software development projects, including:
Overall, CodeScene can help you optimize your codebase for maintainability, readability, and team productivity, which can ultimately lead to a better end product.
Me: “And how does it work?”
ChatGPT: “CodeScene works by analyzing the history of your codebase and using machine learning algorithms to identify patterns and trends in the code. Here are the steps that CodeScene typically follows:
CodeScene can be used as part of a continuous integration and delivery (CI/CD) pipeline, to ensure that code quality is continuously monitored and improved over time. It integrates with a variety of code repositories, including Git, GitHub, and Bitbucket, and can be used with a variety of programming languages, including Java, C#, and Python.”
Now I have an Idea of what benefits that tool can bring to my project and how this works. This can be a base for making a decision to try or not to try this tool. If I’m not sure about that for now, I can go on asking. If this leads me to a tool that helps me handle my issues, ChatGPT would have had a positive influence on my project quality. And that is likely because I used it for a purpose that it was designed for. And it is very easy if you use it in that way. And even if it’s possible to let ChatGPT do things that it is not specified, like writing scripts, I would prefer to use it mainly for research. And this research can be, that I’m looking for AI-powered tools, that can improve my work in the field of software quality assurance.
It seems like 2023 is the start of a new AI era. So let’s prepare ourselves for the future and learn to use the new generation of Software QA Tools! And of course, using chatbots can help us find the right tool that fulfills our needs.
Good Luck! And reach out if we can help you in any way!