Case study: Using LLMs to assist with solving programming homework assignments

Abstract

Nowadays, students have the option of using LLMs for assistance in solving homework assignments. Moreover, most LLMs, like ChatGPT, are also trained on large sets of source code and thus can be used to assist in programming exercises. In this paper, we present a case study based on data collected over the course of 1.5 semesters, where students of three programming-related courses were explicitly permitted to use such models while solving homework assignments. In a qualitative evaluation, we observe that there might be a difference between targeted requests for an answer to specific questions and requests for a complete solution from an LLM. Particularly, targeted requests might be pedagogical feasible and enhance the learning experience. Additionally, we discuss the potential of LLM applications in programming education, with a focus on the intermediate level and beyond.