Automated API Testing

APIs, or Application Programming Interfaces, are the connectors that allow different software systems to communicate. They establish a standardized communication protocol for sending and receiving data, ensuring consistency across platforms. The OpenAPI Specification (OAS) is a popular format used to document these APIs in a machine-readable way (typically in JSON or YAML). Because it’s both comprehensive and machine-readable, OAS powers many automation scenarios in API testing, acting as a blueprint for test generation. 

The Flow of Automated API Testing 

At System Verification, our team has developed an end-to-end workflow for automated API testing. In this post, we will walk you through each stage of our process, emphasizing how we leverage prompt engineering and Large Language Models (LLMs) to generate and execute tests in a structured manner.

The diagram above outlines the steps involved in automated API testing. They are as follows: 

1. Requirements and Specification 

• Business Requirements: The process starts with understanding the business or technical needs from the development team or the customer. These requirements shape what the API is supposed to do.
• OpenAPI Specification (OAS): It defines everything about the APIs – the available endpoints, request and response formats, data types, and authentication methods. 

• API Models & Inter-Dependencies: In this step, we extract API models from the OpenAPI Specification. This extraction generates detailed representations for each endpoint, capturing essential details such as expected responses, request parameters, and potential constraints. Simultaneously, we extract dependencies among endpoints, which are crucial for generating comprehensive system tests that accurately simulate interactions and data flows within the individual endpoints. 

• Prompt Engineering & LLMs: In this phase, we create detailed prompts that integrate API models, identified endpoint dependencies, and the test data. These comprehensive prompts supply the LLMs with all the contextual information needed to generate accurate test cases, which are then output as structured JSON test models. 

• Refinement: To ensure these test models are accurate, complete, and aligned with real-world scenarios, we employ an additional verification step. This involves crafting a follow-up prompt that rigorously checks that the initially generated test models comply with all required specifications and constraints. 

• Contract Testing: We validate that each API endpoint’s responses match the “contract” defined in the OAS. Our test models compare actual responses against the expected schema, data types, and constraints, ensuring consistent behavior. 

• System Testing: Beyond verifying individual endpoints, we assess interactions among multiple endpoints or services. Our system test models simulate real-world scenarios, verifying that dependencies and data flows work seamlessly in an integrated environment. 

• Framework and Tests: The Test Automation Framework is responsible for converting JSON test models into executable test cases by transforming the structured data into scripts or code that run in the testing environment. This framework can either be manually implemented in the preferred programming language or automatically generated alongside the tests, providing a centralized platform for execution and management. 

• Test Execution & Reporting: Once the test cases are executable, they are run within the Test Automation Framework. After execution, a Test Report is generated, summarizing the outcomes of the tests, highlighting any issues or discrepancies, and providing insights into API reliability and performance. 

• The feedback is then used to adjust our prompts, continuously refining the test models to better match the expected API behavior.
   

Can We Reliably Generate Test Cases from API Specifications? 

The short answer: Yes, we can generate test cases automatically and reliably from a well-defined OAS. These tests are especially effective for contract testing—ensuring the API adheres to the agreed-upon request/response formats. Furthermore, by modeling endpoint interactions, our approach extends to comprehensive system testing. 

Key advantages: 

• Well-defined scope: Because the tests come directly from the specification, you test exactly what’s outlined.
• Simplicity and transparency: Automated tests are straightforward for humans to review, fostering trust and clarity in the testing process. 

Requirements and Limitations 

• A well-written, complete OAS is essential. Inadequate specifications lead to incomplete or misleading test cases. Invest the time to ensure your OAS thoroughly describes every endpoint, parameter, and response. 

• While automation handles a significant portion of the work, human oversight is crucial to catch nuances. Automated tests can misinterpret ambiguous specifications or fail to cover unusual edge cases. 

• Automated tests might flag failures, but diagnosing the root cause can be challenging. Is the API truly broken, or was the test itself incorrectly generated or configured? Without careful analysis, you risk false positives (flagging valid behavior as a bug) or missing actual issues. 

Conclusion 

Automated API testing using the OpenAPI Specification can significantly enhance both efficiency and accuracy. At System Verification, we’ve developed a comprehensive flow—from extracting and modeling the APIs to generating and refining tests with LLMs—that ensures thorough validation against the defined contract. While automation can handle a great deal of the workload, human expertise remains critical for ensuring the highest quality. By striking the right balance between machine-driven and human-led efforts, you can deliver robust, reliable APIs that meet evolving business and technical demands. 

Want to learn more about what we do within AI?