Specification and evaluation of multi-agent LLM systems - prototype and cybersecurity applications
Lade...
Autor:innen
Autor:in (Körperschaft)
Publikationsdatum
2025
Typ der Arbeit
Studiengang
Typ
04B - Beitrag Konferenzschrift
Herausgeber:innen
Herausgeber:in (Körperschaft)
Betreuer:in
Übergeordnetes Werk
2025 International Conference on Cybersecurity and AI-Based Systems (Cyber-AI)
Themenheft
DOI der Originalpublikation
Link
Zugehörige Forschungsdaten
Reihe / Serie
Reihennummer
Jahrgang / Band
Ausgabe / Nummer
Seiten / Dauer
340-347
Patentnummer
Verlag / Herausgebende Institution
IEEE
Verlagsort / Veranstaltungsort
Varna
Auflage
Version
Programmiersprache
Abtretungsempfänger:in
Praxispartner:in/Auftraggeber:in
Zusammenfassung
Recent advancements in LLMs indicate potential for novel applications, as evidenced by the reasoning capabilities in the latest OpenAI and DeepSeek models. To apply these models to domain-specific applications beyond text generation, LLM-based multi-agent systems can be utilized to solve complex tasks, particularly by combining reasoning techniques, code generation, and software execution across multiple, potentially specialized LLMs. However, while many evaluations are performed on LLMs, reasoning techniques, and applications individually, their joint specification and combined application are not well understood. Defined specifications for multi-agent LLM systems are required to explore their potential and suitability for specific applications, allowing for systematic evaluations of LLMs, reasoning techniques, and related aspects. This paper reports the results of exploratory research on (1.) multi-agent specification by introducing an agent schema language and (2.) the execution and evaluation of the specifications through a multi-agent system architecture and prototype. The specification language, system architecture, and prototype are first presented in this work, building on an LLM system from prior research. Test cases involving cybersecurity tasks indicate the feasibility of the architecture and evaluation approach. As a result, evaluations could be demonstrated for question answering, server security, and network security tasks completed correctly by agents with LLMs from OpenAI and DeepSeek.
Schlagwörter
Fachgebiet (DDC)
Veranstaltung
IEEE International Conference on Cybersecurity and AI-Based Systems (IEEE Cyber-AI)
Startdatum der Ausstellung
Enddatum der Ausstellung
Startdatum der Konferenz
Enddatum der Konferenz
Datum der letzten Prüfung
ISBN
979-8-3315-6631-9
979-8-3315-6632-6
979-8-3315-6632-6
ISSN
Sprache
Englisch
Während FHNW Zugehörigkeit erstellt
Ja
Zukunftsfelder FHNW
Publikationsstatus
Veröffentlicht
Begutachtung
peer-reviewed
Open Access-Status
Closed
Lizenz
Zitation
Härer, F. (2025). Specification and evaluation of multi-agent LLM systems - prototype and cybersecurity applications. In P. Zahariev, Y. Tashtoush, & O. Darwish (Eds.), 2025 International Conference on Cybersecurity and AI-Based Systems (Cyber-AI) (pp. 340–347). IEEE. https://doi.org/10.1109/Cyber-AI66431.2025.11233474