Smart tools, smarter students. A study on ChatGPT's effects on learning outcomes and exercise quality in business education

Abstract

In education, there's a growing interest in using artificial intelligence, particularly large language models (LLM) such as ChatGPT, to assist and improve learning. Within the context of a Swiss University of applied Sciences employing a flipped classroom methodology and in class exercises, a noticeable shift in student behaviour has been observed. Students overly rely on these tools to complete exercises, diverting their focus away from the exercise content and goals, leading to a decline in the quality of outcomes. It is suspected that they often lack the essential skills to critically assess information accuracy of AI responses and contextualize it within the educational exercise.

This paper aims to evaluate the implications of employing ChatGPT in educational exercises. We aim to assess ChatGPT's proficiency in solving such exercises and to illustrate the knowledge prerequisites for formulating effective prompts. We focus on bachelor students in business education, particularly in project management and business process management. For instance, we will analyse various types of prompts and the resulting outcomes from a project planning exercise conducted by students based on an interview with the project sponsor.

The research methodology involves an analysis of the results obtained, emphasising on the impact on the learning process. This study aims to review the viability of ChatGPT as a tool for solving educational exercises. Additionally, we investigate potential shortcomings and explore required prompts necessary for achieving good outcomes. Moreover, we examine the broader implications of utilising ChatGPT in educational contexts, reflecting on its effects on the learning process. Specifically, exploring ways educators can enhance student learning through effective exercises with the increasing utilisation of AI in the classroom.

This study contributes to the ongoing dialogue on using AI in education by offering insights into the challenges and opportunities associated with ChatGPT's role in solving educational exercises. By addressing the quality of prompts and the subsequent impact on learning, we aim to provide practical guidance for educators and researchers seeking to harness the potential of AI in educational settings.