Automated assessment of programming assignments, also called Auto-Grading, is in widespread use wherever programming courses are offered. Since 2013, UCSB has been using an Auto-Grader developed by Bryce Boe known as submit.cs. Like many other Auto-Graders, submit.cs was developed primarily to investigate a set of research questions; considerations such as long-term support, scaling, infrastructure, and ease of use for instructors were secondary or tertiary concerns.
This project report builds on previous work in a M.S. project report by Hunter Laux to develop a replacement for submit.cs. A key design principle of Laux's work was to leverage existing technologies and APIs wherever possible to reduce the amount of custom code that would have to written, tested, and supported. As such, Laux replaced many of features of submit.cs with features already built into git and GitHub (both the github.com version and the GitHub Enterprise version available at UCSB.)
This work takes this a step further, by replacing additional custom code with features from Jenkins, a free and open-source automation platform that is widely used in industry for continuous integration and deployment.
The result is an Auto-Grader that is flexible, modular, and scalable; one where the amount of application code that must be supported is greatly reduced. Most of the core features of the system are provided by git, GitHub, and Jenkins, with the application code providing routing of information between components. We have designed the system to be backwards compatible, to the extent that assignments developed for submit.cs can be automatically converted into a format that is directly usable in the new system, without modification. We have also introduced several improvements in the workflow for instructors that want to adapt and reuse previous assignments.