Comparative analysis of generative AI models in educational exercise performance

Abstract

The integration of artificial intelligence (AI) in educational settings presents great opportunities for enhancing learning experiences. Within the context of a Swiss University of applied Sciences employing a flipped classroom methodology and in class exercises, a noticeable shift in student behaviour has been observed. The quality of using generative AI for solving case study based exercises in Business Process and Project Management was analysed using OpenAI’s ChatGPT Model 3.5. However, it is unknown how the different models are performing compared to each other when solving exercises in business education. This paper aims to extend the discourse of solving class exercises by conducting a comparative analysis of various generative AI models in the context of educational exercises within the flipped classroom setting, particular in Business Process and Project Management. The study systematically assesses the performance of different AI models, such as GPT-4, BERT, BART, T5, LLM API, etc, in answering selected exercises derived from real-world business scenarios.

This study analyzes the accuracy, relevance, completeness, and contextual understanding exhibited by each AI model in response to a series of exercises. These exercises are designed to mimic real-world business challenges in Business Process and Project Management, thereby providing a meaningful evaluation of each model's use in an educational context. The study further delves into the nuances of prompt construction, examining how variations in prompt design influence the performance of AI models, thereby shedding light on the critical role of effective communication in leveraging AI for educational purposes.

The findings of this research provide educators, researchers, and practitioners with a comprehensive understanding of the comparative strengths and weaknesses of various generative AI models in the context of business education, particularly in Business Process and Project Management. By highlighting key differences in selecting and deploying AI tools for educational exercises, the paper aims to contribute insights into the optimization of AI-assisted learning environments.