Open Access Open Access  Restricted Access Subscription Access

TESTLY AI: AN LLM-ASSISTED MULTI-AGENT FRAMEWORK FOR AUTOMATED WEB APPLICATION TESTING AND BUG ANALYSIS

Ms. A. KAVYA SREE, Ms. S. SHANMATHI

Abstract


Software testing is essential for maintaining the reliability and security of web applications, but manual test design and execution can be repetitive, time-consuming, and difficult to scale. This paper presents Testly AI, an LLM-assisted multi-agent framework for automated web application testing. The proposed system accepts a target URL through a React-based user interface and processes it through a Python/FastAPI backend. Five specialized agents perform planning, test-case generation, test execution, bug analysis, and report generation. The planning stage retrieves the target HTML using HTTP requests and parses its structure with BeautifulSoup. The generator and bug-analysis stages can use Google's Gemini 1.5 Flash large language model when an API key is available; deterministic rule-based fallbacks are provided when the model is unavailable. Selenium is used for browser automation, while ReportLab is used to produce PDF reports and Pillow supports screenshot handling. Test history and website metadata are persisted in a JSON-based store. The project does not use retrieval-augmented generation (RAG), a separately trained machine-learning model, or a dedicated MLOps pipeline. Project execution history contained 16 recorded runs comprising 80 test cases, of which 62 passed and 18 failed, corresponding to an overall observed pass rate of 77.5%. These observations demonstrate the functional feasibility of the multi-agent testing workflow, while also highlighting the need for larger controlled evaluations and ground-truth bug datasets in future work.

 


Full Text:

PDF

References


SeleniumHQ, “Selenium Documentation,” Selenium Project. Available: https://www.selenium.dev/documentation/

FastAPI, “FastAPI Documentation,” Available: https://fastapi.tiangolo.com/

Google, “Gemini API Documentation,” Google AI for Developers. Available: https://ai.google.dev/gemini-api/docs

Beautiful Soup Documentation, “Beautiful Soup Documentation,” Available: https://www.crummy.com/software/BeautifulSoup/bs4/doc/

Pallets, “Requests: HTTP for Humans,” Available: https://requests.readthedocs.io/

ReportLab, “ReportLab User Guide,” Available: https://docs.reportlab.com/

Python Software Foundation, “Python Documentation,” Available: https://docs.python.org/3/

R. S. Pressman and B. R. Maxim, Software Engineering: A Practitioner's Approach, 9th ed., McGraw-Hill, 2019.

M. Hardt et al., “Natural Language Processing and Large Language Models for Software Engineering: A Survey,” for background on language-model-assisted software engineering. Use the final bibliographic record required by the selected journal before submission.


Refbacks

  • There are currently no refbacks.