AWS Extends QA Studio with Test Suites and CI/CD CLI for Automated Regression Testing

AWS Extends QA Studio with Test Suites and CI/CD CLI for Automated Regression Testing

AWS has released Part 2 of QA Studio, extending its agentic QA automation reference solution with test suite management and command-line interface (CLI) integration for continuous integration and continuous delivery (CI/CD) pipelines.

Test Suite Parallel Execution

QA Studio now supports grouping individual use cases into test suites that execute as a batch. According to AWS, each use case within a suite runs independently on its own Amazon ECS on AWS Fargate worker task, enabling concurrent execution rather than sequential processing. A suite of 20 tests can run simultaneously rather than one after another, reducing total execution time.

Test suites are created through the QA Studio web interface by providing a name, description, and optional tags. Each use case maintains its own configuration including starting URL, variables, secrets, and headers. Suite execution pages provide aggregate status showing how many tests succeeded, failed, or are still running, with drill-down capability to review trajectory logs, screenshots, and session recordings for failed tests.

CI/CD Integration via Command-Line Interface

The new QA Studio CLI (qa-studio) enables pipeline integration by running tests locally on CI/CD runners using Amazon Nova Act, rather than dispatching to remote Fargate workers. Results are reported back to the QA Studio deployment.

The CLI supports OAuth 2.0 client credentials authentication through environment variables, eliminating the need for interactive browser login in automated environments. Required scopes include api/suite.read, api/suite.write, api/executions.read, api/executions.write, api/usecases.read, and api/usecases.execute.

Environment and Variable Overrides

The CLI provides several mechanisms for runtime configuration without modifying stored test definitions:

• --base-url flag replaces the domain while preserving path and query parameters, enabling the same test to target development, staging, or production
• --var flag overrides template variables referenced using {{VariableName}} syntax in test steps
• --region flag controls which AWS Region the browser runs in
• --model-id selects the Amazon Nova Act model version

Secrets are stored in AWS Secrets Manager and retrieved at runtime. According to AWS, secret values are not written to execution logs or history records.

Pipeline Integration Features

The CLI uses three-state exit codes for pipeline orchestration:

• Exit code 0: All tests passed (pipeline continues)
• Exit code 1: One or more tests failed (pipeline fails due to test failure)
• Exit code 2: CLI error such as authentication or configuration issues (pipeline fails due to infrastructure error)

Output formatting supports both JSON for programmatic consumption and human-readable format for pipeline logs. During execution, the CLI creates execution records in QA Studio, updates step statuses in real time, and uploads artifacts including trajectory logs and session recordings.

Installation

The QA Studio CLI is available in the project's GitHub repository. Installation requires Python with optional runner dependencies:

pip install -e "./qa-studio-cli[runner]"

What This Means

This release transforms QA Studio from an interactive testing tool into a production CI/CD component. The parallel execution architecture addresses a core limitation of sequential agent-based testing, while the CLI enables automated deployment gates without requiring web interface interaction. The three-state exit code model provides the granularity needed to distinguish test failures from infrastructure issues in pipeline logic. AWS positions this as a reference implementation rather than a managed service, meaning teams must deploy and maintain their own QA Studio infrastructure.