Towards a sustainable operating concept of LLM generated examination support in higher technical education

Abstract

At the University of Applied Science Northwestern Switzerland (FHNW) it can be observed that the usage of preferred learning material shifted in the past years from books and articles to videos, summaries, flashcards and test exams. In a specialization module of the programme BSc Business Information Technology (BIT) at FHNW a Python-based prototype was created to semi-automatically generate MCQ (Multiple Choice Question) based exams and summaries using the - since 2023 - very popular large language models (LLM). The current prototype software is based on the Python libraries langchain for the LLM framework, and streamlit for the user interface. As vector database ChromaDB is used and as LLM Vertex AI from Google. Currently the software is tested with selected documents in order to find out if the quality of generated exams and summaries match the expectations of the lecturers and students. The results will help to answer important questions regarding the practical usage of LLM in education in general. Currently the AI community is flooded with an immense number of software prototypes for nearly every domain. However, a sustainable operational concept for AI in general and LLM in particular is rarely discussed. Our contribution seeks to address this issue for the domain of examination support in higher technical education. Specifically, the following topics are discussed: prompt engineering, testing, quality management, confidentiality assurance and didactic strategy. These issues have to be answered before using the software in a broader context, namely other modules and programmes.