Amazon Nova Act Brings Vision-Based Web Navigation to UX Testing, No Hard-Coded Scripts Required

AWS has released a cloud-deployed UX testing platform built on Amazon Nova Act, a multimodal foundation model that navigates web interfaces through visual understanding rather than hard-coded element selectors. The solution is available as open source through the aws-samples GitHub repository.

How It Works

Unlike Selenium or Playwright, which rely on predefined element selectors that break with interface changes, Nova Act processes screenshots to understand page layout and identify interactive elements through visual cues. According to AWS, the model "analyzes screenshots of web pages just as a human tester would" and makes contextual decisions about navigation.

The system generates test scenarios by processing documentation with Claude 4.5 Sonnet in Amazon Bedrock. For a given task like "purchase a coffee maker via search," the system retrieves relevant information from a knowledge base, then generates detailed test instructions at multiple granularity levels. Nova Act agents execute these flows in parallel browser sessions on Amazon ECS with AWS Fargate.

Architecture Components

The solution comprises four layers:

Documentation Processing: Amazon S3 stores site documentation and user guides. AWS Lambda uses Claude 4.5 Sonnet to transform user flows into testing scenarios after ingesting documentation into an Amazon Bedrock Knowledge Base for semantic search.

Orchestration: Amazon DynamoDB stores generated test flows. DynamoDB Streams triggers batch processing when new flows are available. Lambda functions coordinate test execution and spin up ECS tasks for parallel processing.

Execution: Amazon ECS with Fargate provides serverless compute for parallel test execution. Nova Act agents run in parallel browser sessions with real-time interaction logging.

Analysis: Amazon S3 stores chain-of-thought reasoning logs, screenshots, and behavioral data. Lambda processes results using Amazon Bedrock to analyze execution patterns, calculate usability scores, and identify friction points. Results are stored in DynamoDB and presented through a React dashboard.

Test Flow Creation

The system supports automatic generation from documentation, manual flow definition, or a hybrid approach. AWS recommends the hybrid method: using automatic generation for baseline coverage from existing documentation, then supplementing with manually defined flows for specific test cases or edge scenarios.

For automatic generation, users identify top user flows and provide them as input tasks. The system processes each task by consulting the knowledge base to understand site-specific implementation details, then produces testing instructions across three detail levels. A high-level instruction like "purchase a highly-rated stainless steel coffee maker" expands into precise actions including menu navigation, filter application, and checkout sequence.

Deployment Requirements

The solution requires Node.js v20 or newer, npm v10.8 or newer, an AWS account with CDK setup, and a Nova Act API key. Deployment uses AWS CDK to automate infrastructure provisioning.

What This Means

This represents the first production implementation of vision-based web navigation for systematic UX testing at scale. The approach addresses two core problems: manual testing doesn't scale beyond critical paths, and traditional automation requires constant script maintenance. By using visual understanding instead of DOM selectors, Nova Act could maintain test coverage through interface updates that typically break existing automation. The open-source release provides production-ready infrastructure for organizations to deploy this approach immediately, though pricing details for Nova Act usage were not disclosed.