Specification and evaluation of multi-agent LLM systems - prototype and cybersecurity applications
Loading...
Authors
Author (Corporation)
Publication date
2025
Type of student thesis
Course of study
Collections
Type
04B - Conference paper
Editor (Corporation)
Supervisor
Parent work
2025 International Conference on Cybersecurity and AI-Based Systems (Cyber-AI)
Special issue
DOI of the original publication
Link
Related research data
Series
Series number
Volume
Issue / Number
Pages / Duration
340-347
Patent number
Publisher / Publishing institution
IEEE
Place of publication / Event location
Varna
Edition
Version
Programming language
Assignee
Practice partner / Client
Abstract
Recent advancements in LLMs indicate potential for novel applications, as evidenced by the reasoning capabilities in the latest OpenAI and DeepSeek models. To apply these models to domain-specific applications beyond text generation, LLM-based multi-agent systems can be utilized to solve complex tasks, particularly by combining reasoning techniques, code generation, and software execution across multiple, potentially specialized LLMs. However, while many evaluations are performed on LLMs, reasoning techniques, and applications individually, their joint specification and combined application are not well understood. Defined specifications for multi-agent LLM systems are required to explore their potential and suitability for specific applications, allowing for systematic evaluations of LLMs, reasoning techniques, and related aspects. This paper reports the results of exploratory research on (1.) multi-agent specification by introducing an agent schema language and (2.) the execution and evaluation of the specifications through a multi-agent system architecture and prototype. The specification language, system architecture, and prototype are first presented in this work, building on an LLM system from prior research. Test cases involving cybersecurity tasks indicate the feasibility of the architecture and evaluation approach. As a result, evaluations could be demonstrated for question answering, server security, and network security tasks completed correctly by agents with LLMs from OpenAI and DeepSeek.
Keywords
Subject (DDC)
Event
IEEE International Conference on Cybersecurity and AI-Based Systems (IEEE Cyber-AI)
Exhibition start date
Exhibition end date
Conference start date
Conference end date
Date of the last check
ISBN
979-8-3315-6631-9
979-8-3315-6632-6
979-8-3315-6632-6
ISSN
Language
English
Created during FHNW affiliation
Yes
Strategic action fields FHNW
Publication status
Published
Review
peer-reviewed
Open access category
Closed
License
Citation
Härer, F. (2025). Specification and evaluation of multi-agent LLM systems - prototype and cybersecurity applications. In P. Zahariev, Y. Tashtoush, & O. Darwish (Eds.), 2025 International Conference on Cybersecurity and AI-Based Systems (Cyber-AI) (pp. 340–347). IEEE. https://doi.org/10.1109/Cyber-AI66431.2025.11233474